Nov 03 19:36:15.838758 ip-10-0-133-249 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:36:16.279876 ip-10-0-133-249 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:36:16.279876 ip-10-0-133-249 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:36:16.279876 ip-10-0-133-249 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:36:16.279876 ip-10-0-133-249 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:36:16.285060 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.284980    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:16.285060 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285060    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285065    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285069    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285072    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285075    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285078    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285082    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285085    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285087    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285090    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285092    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285095    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285098    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285100    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285103    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285105    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285111    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285114    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285117    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285120    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:16.285126 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285122    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285125    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285128    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285130    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285133    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285136    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285139    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285142    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285146    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285151    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285155    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285158    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285161    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285163    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285166    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285169    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285171    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285174    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285176    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:16.285595 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285179    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285181    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285184    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285187    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285190    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285192    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285195    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285197    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285200    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285202    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285205    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285207    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285210    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285212    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285218    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285222    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285225    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285228    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285230    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:16.286031 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285233    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285236    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285238    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285241    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285244    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285246    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285249    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285251    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285254    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285256    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285259    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285261    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285264    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285266    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285269    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285272    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285275    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285277    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285280    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:16.286495 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285283    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285286    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285288    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285291    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285294    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285296    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285299    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285302    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285304    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285308    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285310    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285313    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285315    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285318    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285320    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285323    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285326    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285328    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285331    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285334    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:16.286942 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285337    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285339    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285343    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285345    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285348    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285351    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285353    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.285356    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286172    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286185    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286188    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286190    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286193    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286196    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286198    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286201    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286204    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286206    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286211    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:16.287442 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286216    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286218    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286221    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286224    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286227    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286229    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286232    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286235    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286237    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286240    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286242    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286245    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286247    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286250    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286252    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286255    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286257    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286260    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286263    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286265    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:16.287901 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286268    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286270    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286273    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286279    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286282    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286284    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286287    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286290    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286292    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286295    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286297    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286300    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286304    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286306    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286309    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286312    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286314    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286317    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286319    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286322    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:16.288370 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286324    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286327    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286329    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286332    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286334    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286337    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286339    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286342    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286345    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286347    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286350    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286352    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286354    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286357    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286359    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286363    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286371    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286375    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286377    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286380    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:16.288855 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286382    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286385    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286387    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286390    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286393    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286395    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286397    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286400    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286402    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286405    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286407    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286410    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286412    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286415    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286417    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286420    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286422    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286425    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286427    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:16.289316 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286430    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286432    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286435    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286446    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286449    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286452    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286457    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286461    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286464    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286468    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286472    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286475    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286477    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286492    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286495    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.286498    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287027    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287032    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287035    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:16.289794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287037    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287040    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287043    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287046    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287054    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287057    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287059    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287062    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287065    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287068    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287070    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287073    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287075    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287078    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287080    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287083    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287086    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287088    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287091    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:16.290244 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287093    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287096    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287099    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287102    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287110    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287113    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287116    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287119    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287121    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287124    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287127    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287129    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287132    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287134    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287137    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287139    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287142    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287144    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287147    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287149    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:16.290722 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287152    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287154    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287156    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287159    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287161    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287164    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287166    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287169    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287171    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287174    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287176    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287179    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287182    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287184    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287187    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287189    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287192    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287200    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287203    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287206    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:16.291191 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287213    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287216    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287218    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287221    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287223    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287226    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287228    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287231    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287235    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287238    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287241    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287243    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287246    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287248    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287251    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287254    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287256    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287258    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287261    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:16.291678 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287263    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287266    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287268    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287271    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287274    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287276    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287279    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287283    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287286    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287289    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287292    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287301    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287304    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287306    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287309    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287311    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287314    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287316    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287319    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:16.292149 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287321    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287324    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287326    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287330    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287332    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.287335    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288331    2579 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288341    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288352    2579 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288356    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288361    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288365    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288369    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288374    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288377    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288380    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288384    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288391    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288394    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288397    2579 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288400    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288403    2579 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288406    2579 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:36:16.292608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288409    2579 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288421    2579 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288430    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288434    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288437    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288440    2579 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288444    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288447    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288450    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288454    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288456    2579 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288460    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288462    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288466    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288468    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288473    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288476    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288488    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288491    2579 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288495    2579 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288497    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288505    2579 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288508    2579 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288511    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:36:16.293141 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288514    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288517    2579 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288528    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288531    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288534    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288537    2579 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288540    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288543    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288546    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288549    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288552    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288557    2579 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288560    2579 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288565    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288568    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288571    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288574    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288577    2579 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288580    2579 flags.go:64] FLAG: --help="false"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288583    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288586    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288589    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288592    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:36:16.293767 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288596    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288599    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288602    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288605    2579 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288608    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288611    2579 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288614    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288617    2579 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288620    2579 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288623    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288626    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288631    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288634    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288637    2579 flags.go:64] FLAG: --lock-file=""
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288640    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288642    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288646    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288652    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288655    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288658    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288661    2579 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288664    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288668    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288671    2579 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288673    2579 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:36:16.294303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288678    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288681    2579 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288685    2579 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288688    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288691    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288693    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288696    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288700    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288703    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288706    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288712    2579 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288716    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288719    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288722    2579 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288725    2579 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288728    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288731    2579 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288734    2579 flags.go:64] FLAG: --port="10250"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288739    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288742    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0680dc56ed20d9c03"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288745    2579 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288748    2579 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288751    2579 flags.go:64] FLAG: --register-node="true"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288754    2579 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288758    2579 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288761    2579 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:36:16.294904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288764    2579 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288766    2579 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288770    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288774    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288777    2579 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288780    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288783    2579 flags.go:64] FLAG: --runonce="false"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288786    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288789    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288792    2579 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288795    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288797    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288801    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288804    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288807    2579 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288810    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288812    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288815    2579 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288818    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288821    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288825    2579 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288827    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288833    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288836    2579 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288840    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:36:16.295566 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288847    2579 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288850    2579 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288852    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288855    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288859    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288862    2579 flags.go:64] FLAG: --v="2"
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288867    2579 flags.go:64] FLAG: --version="false"
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288871    2579 flags.go:64] FLAG: --vmodule=""
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288875    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.288878    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289040    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289044    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289047    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289050    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289053    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289056    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289058    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289061    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289063    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289066    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289069    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289071    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:16.296146 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289074    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289076    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289079    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289082    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289084    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289087    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289089    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289092    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289095    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289099    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289101    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289104    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289107    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289110    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289112    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289115    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289117    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289120    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289122    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:16.296673 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289125    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289127    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289130    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289133    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289135    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289138    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289140    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289143    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289145    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289148    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289150    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289152    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289155    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289158    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289160    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289163    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289166    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289168    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289171    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:16.297213 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289173    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289176    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289178    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289181    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289183    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289186    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289189    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289191    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289194    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289196    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289214    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289218    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289222    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289226    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289230    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289233    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289236    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289239    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289243    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:16.297750 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289245    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289248    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289250    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289253    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289255    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289258    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289261    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289263    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289266    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289269    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289271    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289274    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289276    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289279    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289281    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289284    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289287    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289289    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289292    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289294    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:16.298188 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289297    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289300    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289305    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289308    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289310    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289313    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289315    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289318    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289321    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289323    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289326    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289329    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289332    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289335    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289337    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289340    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:16.298757 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289342    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.289367    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289404    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289421    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289425    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289428    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289432    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289434    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289437    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289440    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289443    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289445    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289448    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289451    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289453    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289456    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:16.299287 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289458    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289461    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289464    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289466    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289469    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289471    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289474    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289476    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289491    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289494    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289497    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289499    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289502    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289505    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289508    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289510    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289512    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289515    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289520    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:16.299882 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289524    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289529    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289532    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289535    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289538    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289541    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289543    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289546    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289548    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289551    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289554    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289556    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289559    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289561    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289564    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289566    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289569    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289571    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289574    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289576    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:16.300392 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289579    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289581    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289584    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289586    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289589    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289591    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289596    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289599    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289603    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289606    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289609    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289613    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289615    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289619    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289622    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289624    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289627    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289630    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289632    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:16.300939 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289635    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289637    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289640    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289643    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289645    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289648    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289650    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289653    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289655    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289658    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289660    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289663    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289666    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289668    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289670    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289673    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289675    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289678    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289680    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289683    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:16.301494 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289686    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289688    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289691    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289693    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289697    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289700    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289704    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289707    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289710    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289712    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289715    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289717    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289720    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.289722    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.296124    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.296140    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.296717    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.296723    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296782    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:16.302016 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296788    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296792    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296795    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296798    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296801    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296803    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296806    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296809    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296812    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296814    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296817    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296820    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296822    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296825    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296828    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296830    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296833    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296836    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296838    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:16.302549 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296841    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296843    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296846    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296848    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296851    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296854    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296858    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296862    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296865    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296870    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296873    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296876    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296878    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296881    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296884    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296886    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296889    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296891    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296894    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:16.303001 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296897    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296900    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296902    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296905    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296907    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296910    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296912    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296914    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296917    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296920    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296922    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296925    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296927    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296930    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296932    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296935    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296937    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296940    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296942    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:16.303444 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296945    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296948    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296950    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296953    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296958    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296962    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296966    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296969    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296972    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296975    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296978    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296981    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296983    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296986    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296988    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296991    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296993    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296996    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.296998    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297001    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:16.303950 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297004    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297006    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297009    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297011    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297014    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297016    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297019    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297022    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297025    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297027    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297030    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297032    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297035    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297038    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297040    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297043    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297046    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297049    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297051    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297054    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:16.304416 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297057    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297059    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297062    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297065    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297067    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297070    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297072    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297075    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.297103    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297237    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297240    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297243    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297246    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297249    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297251    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:16.304894 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297255    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297259    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297262    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297265    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297268    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297271    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297273    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297276    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297278    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297281    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297283    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297285    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297288    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297291    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297293    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297296    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297299    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297301    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297304    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:16.305263 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297306    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297309    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297311    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297314    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297316    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297319    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297321    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297324    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297326    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297329    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297331    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297334    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297336    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297339    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297341    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297344    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297346    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297349    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297351    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297354    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:16.305731 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297356    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297359    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297361    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297364    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297366    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297369    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297371    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297373    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297376    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297379    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297381    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297384    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297386    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297389    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297391    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297394    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297397    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297399    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297402    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297404    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:16.306200 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297406    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297409    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297412    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297414    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297416    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297419    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297421    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297424    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297427    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297429    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297432    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297434    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297437    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297439    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297442    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297444    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297446    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297449    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297452    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297454    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:16.306702 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297457    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297459    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297464    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297467    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297469    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297472    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297474    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297477    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297495    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297497    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297500    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297502    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297505    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297508    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297510    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297513    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297515    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297518    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297520    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:16.307152 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297523    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297526    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.297549    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297581    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297585    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297588    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297591    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297594    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297596    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297599    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297601    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297604    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297607    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297609    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297612    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:16.307626 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297614    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297618    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297622    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297625    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297627    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297630    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297633    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297635    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297638    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297640    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297643    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297645    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297648    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297650    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297653    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297655    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297660    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297663    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297665    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:16.307986 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297668    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297671    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297673    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297676    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297679    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297683    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297686    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297688    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297691    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297694    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297697    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297699    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297702    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297704    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297707    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297710    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297712    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297715    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297718    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297720    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:16.308447 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297723    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297725    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297728    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297730    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297733    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297735    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297738    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297740    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297743    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297747    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297749    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297752    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297755    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297757    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297761    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297763    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297766    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297770    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297772    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297775    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:16.308995 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297778    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297780    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297783    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297785    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297788    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297790    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297793    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297795    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297798    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297801    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297804    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297806    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297809    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297811    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297814    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297816    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297819    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297821    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297824    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297826    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:16.309519 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297829    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297833    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297835    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297839    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297843    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297846    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297849    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297851    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297854    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297858    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297860    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297864    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297867    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297869    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:16.297872    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.297927    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.301804    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.303087    2579 server.go:1014] "Starting client certificate rotation"
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.303173    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:36:16.309982 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.303871    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:36:16.310425 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.305303    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:36:16.310425 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.307244    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:36:16.320831 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.320812    2579 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:36:16.324794 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.324773    2579 log.go:25] "Validated CRI v1 image API"
Nov 03 19:36:16.325623 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.325600    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:36:16.329159 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.329136    2579 fs.go:135] Filesystem UUIDs: map[2cb1254e-94a4-4d99-ae76-a2a48d1ddd11:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c503ccbe-cc5f-48b3-860d-24aa5b9a5e86:/dev/nvme0n1p4]
Nov 03 19:36:16.329226 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.329158    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 19:36:16.334852 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.334738    2579 manager.go:217] Machine: {Timestamp:2026-11-03 19:36:16.333413103 +0000 UTC m=+0.349992131 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092731 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f62831c0fb56fe29035638d88aaea SystemUUID:ec2f6283-1c0f-b56f-e290-35638d88aaea BootID:27ecc72c-c281-463e-a55c-4a8b2a0976c2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cd:ea:19:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cd:ea:19:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:f4:ad:03:37:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:36:16.334852 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.334844    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:36:16.334989 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.334919    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:36:16.337394 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.337370    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:36:16.337584 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.337393    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-249.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:36:16.338177 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.338166    2579 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:36:16.338214 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.338177    2579 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:36:16.338214 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.338204    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:36:16.338267 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.338223    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:36:16.338925 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.338914    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:36:16.339180 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.339159    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:36:16.343598 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.343586    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:36:16.343710 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.343701    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:36:16.343749 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.343736    2579 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:36:16.343749 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.343745    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:36:16.343806 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.343754    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:36:16.343806 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.343763    2579 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:36:16.343806 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.343777    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:36:16.345913 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.345891    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:36:16.348465 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.348449    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:36:16.352827 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.352811    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:36:16.353618 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.353606    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:36:16.353668 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.353637    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:36:16.353807 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.353797    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:36:16.353840 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.353814    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:36:16.353840 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.353820    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:36:16.353840 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.353826    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:36:16.353840 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.353832    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:36:16.353840 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.353838    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:36:16.354387 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.354379    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:36:16.354415 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.354389    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:36:16.354415 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.354395    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:36:16.354415 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.354402    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:36:16.354415 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.354410    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:36:16.354581 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.354419    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:36:16.354991 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.354980    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:36:16.355024 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.354994    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:36:16.358773 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.358760    2579 server.go:1263] "Started kubelet"
Nov 03 19:36:16.358904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.358848    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:36:16.359067 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.359005    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:36:16.359125 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.359089    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:36:16.359477 ip-10-0-133-249 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:36:16.360152 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.360134    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:36:16.360651 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.360613    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-249.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 19:36:16.361091 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.361079    2579 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:36:16.364060 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.364043    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q67h9"
Nov 03 19:36:16.364422 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.364396    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-249.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 19:36:16.364422 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.364395    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 19:36:16.364762 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.363889    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-249.us-east-2.compute.internal.18e4a017f532a876  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-249.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-249.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:36:16.358729846 +0000 UTC m=+0.375308873,LastTimestamp:2026-11-03 19:36:16.358729846 +0000 UTC m=+0.375308873,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-249.us-east-2.compute.internal,}"
Nov 03 19:36:16.364851 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.364796    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 19:36:16.366168 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.366147    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:36:16.367056 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.367040    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:36:16.367167 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.367147    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:16.367715 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.367646    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:36:16.367715 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.367652    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:36:16.367715 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.367672    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:36:16.367904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.367749    2579 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:36:16.367904 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.367759    2579 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:36:16.368010 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.367911    2579 factory.go:55] Registering systemd factory
Nov 03 19:36:16.368010 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.367925    2579 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:36:16.368118 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.368105    2579 factory.go:153] Registering CRI-O factory
Nov 03 19:36:16.368164 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.368123    2579 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:36:16.368217 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.368167    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:36:16.368217 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.368181    2579 factory.go:103] Registering Raw factory
Nov 03 19:36:16.368217 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.368191    2579 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:36:16.368941 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.368927    2579 manager.go:319] Starting recovery of all containers
Nov 03 19:36:16.374750 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.374726    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-249.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 19:36:16.375118 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.375098    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 19:36:16.376310 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.376279    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q67h9"
Nov 03 19:36:16.382104 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.382087    2579 manager.go:324] Recovery completed
Nov 03 19:36:16.388565 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.388549    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:16.390473 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.390455    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:16.390575 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.390504    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:16.390575 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.390519    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:16.391052 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.391040    2579 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:36:16.391052 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.391044    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:16.391052 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.391049    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:36:16.391176 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.391063    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:36:16.391790 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.391775    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:16.391853 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.391800    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:16.391853 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.391810    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:16.393249 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.393239    2579 policy_none.go:50] "Start"
Nov 03 19:36:16.393282 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.393253    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:36:16.393282 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.393262    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:36:16.395218 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.395207    2579 policy_none.go:44] "Start"
Nov 03 19:36:16.441062 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.441031    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:36:16.443493 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.443465    2579 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:36:16.444000 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.443984    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:36:16.444058 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.444003    2579 server.go:87] "Starting device plugin registration server"
Nov 03 19:36:16.444288 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.444277    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:36:16.444344 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.444319    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:36:16.444400 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.444384    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:36:16.444513 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.444500    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:36:16.444568 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.444513    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:36:16.445161 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.445145    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:36:16.445253 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.445189    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:16.522070 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.522028    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:36:16.523343 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.523323    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:36:16.523400 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.523348    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:36:16.523400 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.523367    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:36:16.523508 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.523405    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 19:36:16.527736 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.527708    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:36:16.544792 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.544746    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:16.545629 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.545615    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:16.545705 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.545646    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:16.545705 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.545659    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:16.545705 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.545685    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.554834 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.554654    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.554928 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.554845    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-249.us-east-2.compute.internal\": node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:16.578809 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.578782    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:16.624363 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.624308    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"]
Nov 03 19:36:16.624532 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.624395    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:16.625418 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.625399    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:16.625525 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.625434    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:16.625525 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.625450    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:16.626839 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.626827    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:16.626995 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.626979    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.627039 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.627014    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:16.627623 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.627607    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:16.627623 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.627613    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:16.627755 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.627639    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:16.627755 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.627649    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:16.627755 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.627653    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:16.627755 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.627679    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:16.628915 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.628899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.628959 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.628933    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:16.629660 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.629639    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:16.629781 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.629674    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:16.629781 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.629688    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:16.652113 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.652087    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found" node="ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.656421 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.656403    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found" node="ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.669632 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.669615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/583fd7cf27a9a1aaee6d847adb49df9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"583fd7cf27a9a1aaee6d847adb49df9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.669713 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.669639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/583fd7cf27a9a1aaee6d847adb49df9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"583fd7cf27a9a1aaee6d847adb49df9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.669713 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.669659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/465e529db144367f617f146802de7125-config\") pod \"kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"465e529db144367f617f146802de7125\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.679118 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.679099    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:16.770154 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.770124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/583fd7cf27a9a1aaee6d847adb49df9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"583fd7cf27a9a1aaee6d847adb49df9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.770154 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.770154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/583fd7cf27a9a1aaee6d847adb49df9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"583fd7cf27a9a1aaee6d847adb49df9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.770341 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.770172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/465e529db144367f617f146802de7125-config\") pod \"kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"465e529db144367f617f146802de7125\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.770341 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.770241    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/583fd7cf27a9a1aaee6d847adb49df9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"583fd7cf27a9a1aaee6d847adb49df9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.770341 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.770240    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/583fd7cf27a9a1aaee6d847adb49df9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"583fd7cf27a9a1aaee6d847adb49df9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.770341 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.770293    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/465e529db144367f617f146802de7125-config\") pod \"kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"465e529db144367f617f146802de7125\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.779198 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.779176    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:16.879988 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.879931    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:16.955202 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.955177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.960575 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:16.960036    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:16.980512 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:16.980475    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:17.081073 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:17.081030    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:17.181887 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:17.181865    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:17.282878 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:17.282824    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:17.303191 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:17.303161    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 19:36:17.303342 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:17.303323    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:36:17.381330 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:17.381270    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:31:16 +0000 UTC" deadline="2028-05-30 03:56:36.481677095 +0000 UTC"
Nov 03 19:36:17.381330 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:17.381319    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13760h20m19.10036261s"
Nov 03 19:36:17.383597 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:17.383577    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:17.484190 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:17.484151    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:17.584392 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:17.584352    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:17.589337 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:17.589313    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:36:17.627176 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:17.627142    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465e529db144367f617f146802de7125.slice/crio-fa4b784afcde48d129fed5739bd18e1912366eb56456f5af792846354f6c0389 WatchSource:0}: Error finding container fa4b784afcde48d129fed5739bd18e1912366eb56456f5af792846354f6c0389: Status 404 returned error can't find the container with id fa4b784afcde48d129fed5739bd18e1912366eb56456f5af792846354f6c0389
Nov 03 19:36:17.627602 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:17.627569    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583fd7cf27a9a1aaee6d847adb49df9a.slice/crio-e4573b6c07d46d668269dd08759155a50b743f1ab4fae0d8f764e63052809f55 WatchSource:0}: Error finding container e4573b6c07d46d668269dd08759155a50b743f1ab4fae0d8f764e63052809f55: Status 404 returned error can't find the container with id e4573b6c07d46d668269dd08759155a50b743f1ab4fae0d8f764e63052809f55
Nov 03 19:36:17.632909 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:17.632890    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:36:17.684880 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:17.684852    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:17.760024 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:17.759942    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:36:17.785932 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:17.785903    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:36:17.804792 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:17.804766    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:36:17.868276 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:17.868237    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:17.882321 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:17.882297    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:36:17.883583 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:17.883568    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:36:17.894166 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:17.894145    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:36:18.346368 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.346336    2579 apiserver.go:52] "Watching apiserver"
Nov 03 19:36:18.354524 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.354497    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:36:18.354988 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.354958    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-5ktfb","openshift-image-registry/node-ca-mmxqm","kube-system/global-pull-secret-syncer-hc967","openshift-multus/multus-additional-cni-plugins-bt6hc","openshift-network-operator/iptables-alerter-5qqvk","openshift-ovn-kubernetes/ovnkube-node-v2dj5","kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal","openshift-multus/multus-k9k97","openshift-multus/network-metrics-daemon-bpk6k","openshift-network-diagnostics/network-check-target-vcw9t","kube-system/konnectivity-agent-6q6gg","openshift-cluster-node-tuning-operator/tuned-vbcxg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"]
Nov 03 19:36:18.358238 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.358214    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.361300 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.361280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:36:18.363139 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.363113    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:36:18.363268 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.363253    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:36:18.363336 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.363328    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:36:18.363399 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.363118    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lrbp4\""
Nov 03 19:36:18.364372 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.364348    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:36:18.364968 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.364936    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:36:18.365076 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.365043    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:36:18.370787 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.370765    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 19:36:18.370945 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.370925    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:36:18.371220 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.371201    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ldllm\""
Nov 03 19:36:18.371446 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.371432    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:36:18.371522 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.371509    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:36:18.371753 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.371729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:36:18.372120 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.372103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.372177 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.372146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:18.372277 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:18.372221    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:18.374002 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.373979    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:36:18.376894 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.376875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.377047 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.377018    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:36:18.377128 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.377059    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbkg2\""
Nov 03 19:36:18.377128 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.377101    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:36:18.377699 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.377681    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:36:18.379100 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.379082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:18.379192 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:18.379137    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:18.379485 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.379441    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bkd9f\""
Nov 03 19:36:18.381803 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.381782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-slash\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.381912 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.381818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-run-netns\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.381912 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.381845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-log-socket\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.381912 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.381872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-kubernetes\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.381912 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.381897    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-cni-bin\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.382128 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.381921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysctl-d\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.382128 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.381951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-var-lib-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.382128 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.381981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-cni-netd\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.382128 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ad627830-62cd-4f5b-bb63-76b5944b91ff-serviceca\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:36:18.382128 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c43e3b1f-9e9c-4768-8c45-e779762180bd-iptables-alerter-script\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:36:18.382128 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j68r2\" (UniqueName: \"kubernetes.io/projected/ad627830-62cd-4f5b-bb63-76b5944b91ff-kube-api-access-j68r2\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:36:18.382128 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-modprobe-d\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.382488 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysconfig\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.382488 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-kubelet\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.382488 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-lib-modules\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.382488 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b6a3864-3172-40f0-9ab4-7327e2bf461c-tmp\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.382488 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-etc-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.382488 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-ovn\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.382488 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-env-overrides\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.382488 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad627830-62cd-4f5b-bb63-76b5944b91ff-host\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:36:18.382488 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-systemd\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.382488 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-var-lib-kubelet\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.382488 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ssv\" (UniqueName: \"kubernetes.io/projected/2b6a3864-3172-40f0-9ab4-7327e2bf461c-kube-api-access-v8ssv\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.382488 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-node-log\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d6f9087-55e4-4e30-905a-3a1493336ded-ovn-node-metrics-cert\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-sys\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-host\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d-agent-certs\") pod \"konnectivity-agent-6q6gg\" (UID: \"16b10ae2-69df-49f1-b0b3-77c98ee4cc3d\") " pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d-konnectivity-ca\") pod \"konnectivity-agent-6q6gg\" (UID: \"16b10ae2-69df-49f1-b0b3-77c98ee4cc3d\") " pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-ovnkube-config\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thm8g\" (UniqueName: \"kubernetes.io/projected/2d6f9087-55e4-4e30-905a-3a1493336ded-kube-api-access-thm8g\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysctl-conf\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-tuned\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c43e3b1f-9e9c-4768-8c45-e779762180bd-host-slash\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttmj2\" (UniqueName: \"kubernetes.io/projected/c43e3b1f-9e9c-4768-8c45-e779762180bd-kube-api-access-ttmj2\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.382883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-systemd-units\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.383009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-systemd\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.383204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.383097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:36:18.384041 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.383405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.384041 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.383453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.384041 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.383495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.384041 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.383522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-ovnkube-script-lib\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.384041 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.383549    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-run\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.384041 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.383592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4ndj\" (UniqueName: \"kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj\") pod \"network-check-target-vcw9t\" (UID: \"124e9f93-f05c-4eee-8054-ac67754550c3\") " pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:18.385113 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.384793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.386731 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.386708    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:36:18.387825 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.387806    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:18.387908 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:18.387879    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:18.390303 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.390282    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.412003 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.411974    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dr4fg"
Nov 03 19:36:18.424783 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.424354    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dr4fg"
Nov 03 19:36:18.468854 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.468828    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:36:18.471179 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.471155    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhpjg\""
Nov 03 19:36:18.471297 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.471200    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:36:18.471297 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.471285    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rs44g\""
Nov 03 19:36:18.471367 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.471353    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndfpg\""
Nov 03 19:36:18.471413 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.471388    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.477942    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.478055    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.478148    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.478225    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.478289    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.478334    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.478395    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.478438    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.478535    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.478697    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.478706    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.478761    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xv4gr\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.478825    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.478831    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzp4r\""
Nov 03 19:36:18.479189 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.479002    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:36:18.484010 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.483979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqjz\" (UniqueName: \"kubernetes.io/projected/658b22ed-1324-4c81-b112-f051fdd3baa4-kube-api-access-lbqjz\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:18.484114 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.484114 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nngxb\" (UniqueName: \"kubernetes.io/projected/4dd8ccdd-1987-4200-8963-983dc4fd445d-kube-api-access-nngxb\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.484114 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-ovnkube-config\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.484269 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysctl-conf\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.484269 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.484269 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.484413 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-multus-certs\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.484413 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-systemd\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.484413 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.484413 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ndj\" (UniqueName: \"kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj\") pod \"network-check-target-vcw9t\" (UID: \"124e9f93-f05c-4eee-8054-ac67754550c3\") " pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:18.484614 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484445    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.484614 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-cni-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.484614 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-cni-multus\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.484614 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-slash\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.484614 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-run-netns\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.484614 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-cni-bin\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.484822 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-cni-bin\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.484822 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysctl-d\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.484822 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-var-lib-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.484822 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c43e3b1f-9e9c-4768-8c45-e779762180bd-iptables-alerter-script\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:36:18.484822 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dd8ccdd-1987-4200-8963-983dc4fd445d-cni-binary-copy\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.484822 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484753    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-ovnkube-config\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.485014 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484895    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-systemd\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.485014 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484934    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-run-netns\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.485014 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484939    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysctl-conf\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.485014 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484939    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysctl-d\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.485014 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484969    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-cni-bin\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.485014 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e79d1e48-bb87-4a1b-9b88-95a1551e959c-tmp-dir\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:36:18.485014 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.484989    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-slash\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.485014 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j68r2\" (UniqueName: \"kubernetes.io/projected/ad627830-62cd-4f5b-bb63-76b5944b91ff-kube-api-access-j68r2\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:36:18.485370 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485030    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-var-lib-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.485370 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysconfig\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.485370 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-lib-modules\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.485370 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae317c95-3690-4603-a062-652384d87f55-dbus\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:18.485370 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-daemon-config\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.485370 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485278    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-lib-modules\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.485370 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-etc-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.485689 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485373    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysconfig\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.485689 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-env-overrides\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.485689 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485406    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-etc-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.485689 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-systemd\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.485689 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ssv\" (UniqueName: \"kubernetes.io/projected/2b6a3864-3172-40f0-9ab4-7327e2bf461c-kube-api-access-v8ssv\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.485689 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485597    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-systemd\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.485881 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d-agent-certs\") pod \"konnectivity-agent-6q6gg\" (UID: \"16b10ae2-69df-49f1-b0b3-77c98ee4cc3d\") " pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:36:18.485881 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-cnibin\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.485881 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-sys\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.485999 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-host\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.485999 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485917    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-sys\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.485999 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-system-cni-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.485999 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.485975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-kubelet\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.486139 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486041    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-host\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.486139 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486047    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-env-overrides\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.486139 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thm8g\" (UniqueName: \"kubernetes.io/projected/2d6f9087-55e4-4e30-905a-3a1493336ded-kube-api-access-thm8g\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.486139 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486050    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c43e3b1f-9e9c-4768-8c45-e779762180bd-iptables-alerter-script\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:36:18.486139 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-tuned\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.486139 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486129    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:36:18.486420 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c43e3b1f-9e9c-4768-8c45-e779762180bd-host-slash\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:36:18.486420 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttmj2\" (UniqueName: \"kubernetes.io/projected/c43e3b1f-9e9c-4768-8c45-e779762180bd-kube-api-access-ttmj2\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:36:18.486420 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-registration-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.486420 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486305    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c43e3b1f-9e9c-4768-8c45-e779762180bd-host-slash\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:36:18.486420 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-device-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.486420 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-hostroot\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.486420 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-systemd-units\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.486651 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486448    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-systemd-units\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.486685 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.486718 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.486761 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-ovnkube-script-lib\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.486761 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-run\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.486858 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-etc-selinux\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.486858 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-system-cni-dir\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.486858 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.487013 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-log-socket\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.487013 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-kubernetes\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.487013 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-sys-fs\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.487013 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn6n4\" (UniqueName: \"kubernetes.io/projected/4af39c8b-d06b-4569-bc27-ca478cc4852c-kube-api-access-wn6n4\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.487013 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.486999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:18.487258 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-cnibin\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.487258 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-conf-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.487258 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-cni-netd\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.487258 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ad627830-62cd-4f5b-bb63-76b5944b91ff-serviceca\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:36:18.487258 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e79d1e48-bb87-4a1b-9b88-95a1551e959c-hosts-file\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:36:18.487258 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-modprobe-d\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.487258 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487216    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-log-socket\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.487544 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-kubelet\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.487544 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b6a3864-3172-40f0-9ab4-7327e2bf461c-tmp\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.487544 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487316    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-kubernetes\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.487544 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-cni-binary-copy\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.487544 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae317c95-3690-4603-a062-652384d87f55-kubelet-config\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:18.487544 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-k8s-cni-cncf-io\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.487544 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-netns\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.487544 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487354    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-cni-netd\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.487544 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28xl5\" (UniqueName: \"kubernetes.io/projected/e79d1e48-bb87-4a1b-9b88-95a1551e959c-kube-api-access-28xl5\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:36:18.487544 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-ovn\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.487544 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad627830-62cd-4f5b-bb63-76b5944b91ff-host\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:36:18.487544 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-var-lib-kubelet\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.488213 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d-konnectivity-ca\") pod \"konnectivity-agent-6q6gg\" (UID: \"16b10ae2-69df-49f1-b0b3-77c98ee4cc3d\") " pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:36:18.488213 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxm9f\" (UniqueName: \"kubernetes.io/projected/fc6cb168-a115-4918-a092-bc340accd868-kube-api-access-zxm9f\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.488213 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:18.488213 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-socket-dir-parent\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.488213 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-etc-kubernetes\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.488213 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-node-log\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.488213 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d6f9087-55e4-4e30-905a-3a1493336ded-ovn-node-metrics-cert\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.488213 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-socket-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.488213 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487943    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-ovnkube-script-lib\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.488213 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487998    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.488213 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.488037    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.488213 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.488174    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-run\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.488921 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.488239    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ad627830-62cd-4f5b-bb63-76b5944b91ff-serviceca\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:36:18.488921 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.488251    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-kubelet\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.488921 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.487261    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-modprobe-d\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.488921 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.488379    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-ovn\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.488921 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.488421    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad627830-62cd-4f5b-bb63-76b5944b91ff-host\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:36:18.488921 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.488471    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-var-lib-kubelet\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.489525 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.489173    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-node-log\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.489633 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.489522    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d-konnectivity-ca\") pod \"konnectivity-agent-6q6gg\" (UID: \"16b10ae2-69df-49f1-b0b3-77c98ee4cc3d\") " pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:36:18.490762 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.490738    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-tuned\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.491116 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.491067    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d-agent-certs\") pod \"konnectivity-agent-6q6gg\" (UID: \"16b10ae2-69df-49f1-b0b3-77c98ee4cc3d\") " pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:36:18.492199 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.492178    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d6f9087-55e4-4e30-905a-3a1493336ded-ovn-node-metrics-cert\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.492439 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.492415    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b6a3864-3172-40f0-9ab4-7327e2bf461c-tmp\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.496082 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:18.494750    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:18.496082 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:18.494776    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:18.496082 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:18.494789    2579 projected.go:196] Error preparing data for projected volume kube-api-access-l4ndj for pod openshift-network-diagnostics/network-check-target-vcw9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:18.496082 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:18.494855    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj podName:124e9f93-f05c-4eee-8054-ac67754550c3 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:18.994833202 +0000 UTC m=+3.011412233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l4ndj" (UniqueName: "kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj") pod "network-check-target-vcw9t" (UID: "124e9f93-f05c-4eee-8054-ac67754550c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:18.497882 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.497854    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j68r2\" (UniqueName: \"kubernetes.io/projected/ad627830-62cd-4f5b-bb63-76b5944b91ff-kube-api-access-j68r2\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:36:18.499815 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.499475    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thm8g\" (UniqueName: \"kubernetes.io/projected/2d6f9087-55e4-4e30-905a-3a1493336ded-kube-api-access-thm8g\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.499815 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.499715    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ssv\" (UniqueName: \"kubernetes.io/projected/2b6a3864-3172-40f0-9ab4-7327e2bf461c-kube-api-access-v8ssv\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.500335 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.500311    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttmj2\" (UniqueName: \"kubernetes.io/projected/c43e3b1f-9e9c-4768-8c45-e779762180bd-kube-api-access-ttmj2\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:36:18.528961 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.528934    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:36:18.531016 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.530965    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal" event={"ID":"465e529db144367f617f146802de7125","Type":"ContainerStarted","Data":"fa4b784afcde48d129fed5739bd18e1912366eb56456f5af792846354f6c0389"}
Nov 03 19:36:18.532458 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.532428    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal" event={"ID":"583fd7cf27a9a1aaee6d847adb49df9a","Type":"ContainerStarted","Data":"e4573b6c07d46d668269dd08759155a50b743f1ab4fae0d8f764e63052809f55"}
Nov 03 19:36:18.588982 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.588941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-cni-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.589170 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.588991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-cni-multus\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.589170 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-cni-bin\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.589170 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dd8ccdd-1987-4200-8963-983dc4fd445d-cni-binary-copy\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.589170 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e79d1e48-bb87-4a1b-9b88-95a1551e959c-tmp-dir\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:36:18.589170 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae317c95-3690-4603-a062-652384d87f55-dbus\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:18.589170 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-daemon-config\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.589476 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-cnibin\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.589476 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-system-cni-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.589476 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-kubelet\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.589476 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-registration-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.589476 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-device-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.589476 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-hostroot\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.589476 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-etc-selinux\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.589476 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-system-cni-dir\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.589476 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.589476 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-sys-fs\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.589476 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6n4\" (UniqueName: \"kubernetes.io/projected/4af39c8b-d06b-4569-bc27-ca478cc4852c-kube-api-access-wn6n4\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-cnibin\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-conf-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e79d1e48-bb87-4a1b-9b88-95a1551e959c-hosts-file\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-cni-binary-copy\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae317c95-3690-4603-a062-652384d87f55-kubelet-config\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-k8s-cni-cncf-io\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-netns\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28xl5\" (UniqueName: \"kubernetes.io/projected/e79d1e48-bb87-4a1b-9b88-95a1551e959c-kube-api-access-28xl5\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxm9f\" (UniqueName: \"kubernetes.io/projected/fc6cb168-a115-4918-a092-bc340accd868-kube-api-access-zxm9f\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-socket-dir-parent\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-etc-kubernetes\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-socket-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqjz\" (UniqueName: \"kubernetes.io/projected/658b22ed-1324-4c81-b112-f051fdd3baa4-kube-api-access-lbqjz\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.590043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nngxb\" (UniqueName: \"kubernetes.io/projected/4dd8ccdd-1987-4200-8963-983dc4fd445d-kube-api-access-nngxb\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.590879 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.590879 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.590879 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.589921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-multus-certs\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.590879 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.590002    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-multus-certs\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.590879 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.590121    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-cni-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.590879 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.590152    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-cni-multus\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.590879 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.590189    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-cni-bin\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.590879 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.590693    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dd8ccdd-1987-4200-8963-983dc4fd445d-cni-binary-copy\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.590963    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-conf-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.590985    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-k8s-cni-cncf-io\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:18.591043    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591047    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-socket-dir-parent\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591082    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e79d1e48-bb87-4a1b-9b88-95a1551e959c-hosts-file\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:18.591118    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret podName:ae317c95-3690-4603-a062-652384d87f55 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:19.091099167 +0000 UTC m=+3.107678202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret") pod "global-pull-secret-syncer-hc967" (UID: "ae317c95-3690-4603-a062-652384d87f55") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591126    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-netns\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591130    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae317c95-3690-4603-a062-652384d87f55-dbus\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591268    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-etc-selinux\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591295    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-etc-kubernetes\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591308    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-cnibin\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591418    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae317c95-3690-4603-a062-652384d87f55-kubelet-config\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591474    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-device-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591508    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e79d1e48-bb87-4a1b-9b88-95a1551e959c-tmp-dir\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:36:18.591529 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591522    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-hostroot\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.592382 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591575    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.592382 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591576    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-system-cni-dir\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.592382 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591637    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-system-cni-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.592382 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591718    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.592382 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591720    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-sys-fs\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.592382 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591748    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-kubelet\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.592382 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:18.591850    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:18.592382 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.591875    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-cnibin\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.592382 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:18.591901    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs podName:658b22ed-1324-4c81-b112-f051fdd3baa4 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:19.091885947 +0000 UTC m=+3.108464979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs") pod "network-metrics-daemon-bpk6k" (UID: "658b22ed-1324-4c81-b112-f051fdd3baa4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:18.592806 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.592438    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-daemon-config\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.592806 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.592493    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-registration-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.592806 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.592559    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.592806 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.592560    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-cni-binary-copy\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.592806 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.592681    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-socket-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.593183 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.593164    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.608036 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.607925    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28xl5\" (UniqueName: \"kubernetes.io/projected/e79d1e48-bb87-4a1b-9b88-95a1551e959c-kube-api-access-28xl5\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:36:18.608881 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.608672    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxm9f\" (UniqueName: \"kubernetes.io/projected/fc6cb168-a115-4918-a092-bc340accd868-kube-api-access-zxm9f\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.608881 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.608835    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nngxb\" (UniqueName: \"kubernetes.io/projected/4dd8ccdd-1987-4200-8963-983dc4fd445d-kube-api-access-nngxb\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.610399 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.610373    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqjz\" (UniqueName: \"kubernetes.io/projected/658b22ed-1324-4c81-b112-f051fdd3baa4-kube-api-access-lbqjz\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:18.611052 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.611031    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6n4\" (UniqueName: \"kubernetes.io/projected/4af39c8b-d06b-4569-bc27-ca478cc4852c-kube-api-access-wn6n4\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.702413 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.702373    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:36:18.707882 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.707836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:18.711223 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.711196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:36:18.713184 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:18.713153    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b10ae2_69df_49f1_b0b3_77c98ee4cc3d.slice/crio-8ef65d4b1d6ce321edb022e352a3ca81c3f22c484c110a9d001c1ae857a2db63 WatchSource:0}: Error finding container 8ef65d4b1d6ce321edb022e352a3ca81c3f22c484c110a9d001c1ae857a2db63: Status 404 returned error can't find the container with id 8ef65d4b1d6ce321edb022e352a3ca81c3f22c484c110a9d001c1ae857a2db63
Nov 03 19:36:18.714536 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.714491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:36:18.719056 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:18.718974    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6f9087_55e4_4e30_905a_3a1493336ded.slice/crio-48cbaded1c0107af1ae8d3c350d4a35a96fbef00ee0ca74b54e3b62b45fff466 WatchSource:0}: Error finding container 48cbaded1c0107af1ae8d3c350d4a35a96fbef00ee0ca74b54e3b62b45fff466: Status 404 returned error can't find the container with id 48cbaded1c0107af1ae8d3c350d4a35a96fbef00ee0ca74b54e3b62b45fff466
Nov 03 19:36:18.722843 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.722819    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:36:18.723418 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:18.723389    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad627830_62cd_4f5b_bb63_76b5944b91ff.slice/crio-677dff29918d74053b715438e63808bad34068d201bb5f1b81f9bbdf43f706b5 WatchSource:0}: Error finding container 677dff29918d74053b715438e63808bad34068d201bb5f1b81f9bbdf43f706b5: Status 404 returned error can't find the container with id 677dff29918d74053b715438e63808bad34068d201bb5f1b81f9bbdf43f706b5
Nov 03 19:36:18.725761 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:18.725736    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6a3864_3172_40f0_9ab4_7327e2bf461c.slice/crio-497e8104f7599dd5143588f908b08e98b54644f90e204a4d1040c78858d2c701 WatchSource:0}: Error finding container 497e8104f7599dd5143588f908b08e98b54644f90e204a4d1040c78858d2c701: Status 404 returned error can't find the container with id 497e8104f7599dd5143588f908b08e98b54644f90e204a4d1040c78858d2c701
Nov 03 19:36:18.731451 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.731317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k9k97"
Nov 03 19:36:18.733910 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:18.733883    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43e3b1f_9e9c_4768_8c45_e779762180bd.slice/crio-0a3ad723292a1d970767de948de47caa2775987494c8849d2c01687519c75cdb WatchSource:0}: Error finding container 0a3ad723292a1d970767de948de47caa2775987494c8849d2c01687519c75cdb: Status 404 returned error can't find the container with id 0a3ad723292a1d970767de948de47caa2775987494c8849d2c01687519c75cdb
Nov 03 19:36:18.739720 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.739694    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:36:18.743721 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:18.743664    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd8ccdd_1987_4200_8963_983dc4fd445d.slice/crio-c6ca5648eb0ccce5b434f08a60c23e27b7fbd452db9df952ad10aa73317cb85a WatchSource:0}: Error finding container c6ca5648eb0ccce5b434f08a60c23e27b7fbd452db9df952ad10aa73317cb85a: Status 404 returned error can't find the container with id c6ca5648eb0ccce5b434f08a60c23e27b7fbd452db9df952ad10aa73317cb85a
Nov 03 19:36:18.750923 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:18.750897    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79d1e48_bb87_4a1b_9b88_95a1551e959c.slice/crio-9347aba7b20c207e2b7a373af25c5a17360da4d60d39de80b696be49f5d36bda WatchSource:0}: Error finding container 9347aba7b20c207e2b7a373af25c5a17360da4d60d39de80b696be49f5d36bda: Status 404 returned error can't find the container with id 9347aba7b20c207e2b7a373af25c5a17360da4d60d39de80b696be49f5d36bda
Nov 03 19:36:18.766700 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.766674    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:36:18.768859 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:18.768829    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:36:18.777786 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:18.777759    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6cb168_a115_4918_a092_bc340accd868.slice/crio-6bb512376fd1e2183d02c940760b7c11982fac4352a61012aa9f93946a6f0d8f WatchSource:0}: Error finding container 6bb512376fd1e2183d02c940760b7c11982fac4352a61012aa9f93946a6f0d8f: Status 404 returned error can't find the container with id 6bb512376fd1e2183d02c940760b7c11982fac4352a61012aa9f93946a6f0d8f
Nov 03 19:36:18.781359 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:36:18.781338    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af39c8b_d06b_4569_bc27_ca478cc4852c.slice/crio-af0d3cf6a78f293142be0e4c8b640931ec1f25ed2946d8db528a4200494f360d WatchSource:0}: Error finding container af0d3cf6a78f293142be0e4c8b640931ec1f25ed2946d8db528a4200494f360d: Status 404 returned error can't find the container with id af0d3cf6a78f293142be0e4c8b640931ec1f25ed2946d8db528a4200494f360d
Nov 03 19:36:19.093905 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.093864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:19.094094 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.093917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:19.094094 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.094032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ndj\" (UniqueName: \"kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj\") pod \"network-check-target-vcw9t\" (UID: \"124e9f93-f05c-4eee-8054-ac67754550c3\") " pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:19.094094 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:19.094051    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:19.094261 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:19.094130    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs podName:658b22ed-1324-4c81-b112-f051fdd3baa4 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:20.0941094 +0000 UTC m=+4.110688415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs") pod "network-metrics-daemon-bpk6k" (UID: "658b22ed-1324-4c81-b112-f051fdd3baa4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:19.094261 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:19.094138    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:19.094261 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:19.094154    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:19.094261 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:19.094164    2579 projected.go:196] Error preparing data for projected volume kube-api-access-l4ndj for pod openshift-network-diagnostics/network-check-target-vcw9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:19.094261 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:19.094182    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:19.094261 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:19.094208    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj podName:124e9f93-f05c-4eee-8054-ac67754550c3 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:20.094196369 +0000 UTC m=+4.110775398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l4ndj" (UniqueName: "kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj") pod "network-check-target-vcw9t" (UID: "124e9f93-f05c-4eee-8054-ac67754550c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:19.094261 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:19.094237    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret podName:ae317c95-3690-4603-a062-652384d87f55 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:20.094221321 +0000 UTC m=+4.110800336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret") pod "global-pull-secret-syncer-hc967" (UID: "ae317c95-3690-4603-a062-652384d87f55") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:19.427483 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.427317    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:31:18 +0000 UTC" deadline="2028-06-13 15:36:50.939538822 +0000 UTC"
Nov 03 19:36:19.427483 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.427381    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14108h0m31.512161768s"
Nov 03 19:36:19.523733 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.523701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:19.523904 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:19.523837    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:19.542492 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.542444    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws" event={"ID":"4af39c8b-d06b-4569-bc27-ca478cc4852c","Type":"ContainerStarted","Data":"af0d3cf6a78f293142be0e4c8b640931ec1f25ed2946d8db528a4200494f360d"}
Nov 03 19:36:19.549702 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.549615    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt6hc" event={"ID":"fc6cb168-a115-4918-a092-bc340accd868","Type":"ContainerStarted","Data":"6bb512376fd1e2183d02c940760b7c11982fac4352a61012aa9f93946a6f0d8f"}
Nov 03 19:36:19.551830 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.551607    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5ktfb" event={"ID":"e79d1e48-bb87-4a1b-9b88-95a1551e959c","Type":"ContainerStarted","Data":"9347aba7b20c207e2b7a373af25c5a17360da4d60d39de80b696be49f5d36bda"}
Nov 03 19:36:19.558305 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.558243    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mmxqm" event={"ID":"ad627830-62cd-4f5b-bb63-76b5944b91ff","Type":"ContainerStarted","Data":"677dff29918d74053b715438e63808bad34068d201bb5f1b81f9bbdf43f706b5"}
Nov 03 19:36:19.563177 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.563063    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5" event={"ID":"2d6f9087-55e4-4e30-905a-3a1493336ded","Type":"ContainerStarted","Data":"48cbaded1c0107af1ae8d3c350d4a35a96fbef00ee0ca74b54e3b62b45fff466"}
Nov 03 19:36:19.567673 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.567592    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6q6gg" event={"ID":"16b10ae2-69df-49f1-b0b3-77c98ee4cc3d","Type":"ContainerStarted","Data":"8ef65d4b1d6ce321edb022e352a3ca81c3f22c484c110a9d001c1ae857a2db63"}
Nov 03 19:36:19.569631 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.569553    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k9k97" event={"ID":"4dd8ccdd-1987-4200-8963-983dc4fd445d","Type":"ContainerStarted","Data":"c6ca5648eb0ccce5b434f08a60c23e27b7fbd452db9df952ad10aa73317cb85a"}
Nov 03 19:36:19.571246 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.571189    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5qqvk" event={"ID":"c43e3b1f-9e9c-4768-8c45-e779762180bd","Type":"ContainerStarted","Data":"0a3ad723292a1d970767de948de47caa2775987494c8849d2c01687519c75cdb"}
Nov 03 19:36:19.575590 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:19.575530    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vbcxg" event={"ID":"2b6a3864-3172-40f0-9ab4-7327e2bf461c","Type":"ContainerStarted","Data":"497e8104f7599dd5143588f908b08e98b54644f90e204a4d1040c78858d2c701"}
Nov 03 19:36:20.102005 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:20.101736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:20.102216 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:20.102038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:20.102216 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:20.102097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ndj\" (UniqueName: \"kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj\") pod \"network-check-target-vcw9t\" (UID: \"124e9f93-f05c-4eee-8054-ac67754550c3\") " pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:20.102344 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:20.102281    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:20.102344 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:20.102301    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:20.102344 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:20.102315    2579 projected.go:196] Error preparing data for projected volume kube-api-access-l4ndj for pod openshift-network-diagnostics/network-check-target-vcw9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:20.102500 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:20.102374    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj podName:124e9f93-f05c-4eee-8054-ac67754550c3 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:22.102355225 +0000 UTC m=+6.118934260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l4ndj" (UniqueName: "kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj") pod "network-check-target-vcw9t" (UID: "124e9f93-f05c-4eee-8054-ac67754550c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:20.102679 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:20.102661    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:20.102747 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:20.102707    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret podName:ae317c95-3690-4603-a062-652384d87f55 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:22.102696529 +0000 UTC m=+6.119275559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret") pod "global-pull-secret-syncer-hc967" (UID: "ae317c95-3690-4603-a062-652384d87f55") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:20.102814 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:20.102759    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:20.102814 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:20.102788    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs podName:658b22ed-1324-4c81-b112-f051fdd3baa4 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:22.10277978 +0000 UTC m=+6.119358809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs") pod "network-metrics-daemon-bpk6k" (UID: "658b22ed-1324-4c81-b112-f051fdd3baa4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:20.428707 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:20.428589    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:31:18 +0000 UTC" deadline="2028-06-02 23:04:56.097008808 +0000 UTC"
Nov 03 19:36:20.428707 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:20.428631    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13851h28m35.668381874s"
Nov 03 19:36:20.528938 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:20.525959    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:20.528938 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:20.526071    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:20.528938 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:20.526425    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:20.528938 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:20.526515    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:20.587004 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:20.586963    2579 generic.go:356] "Generic (PLEG): container finished" podID="583fd7cf27a9a1aaee6d847adb49df9a" containerID="32b1aa6939637b01a067414caad8da0bbeb9e2ddff9e9a2335313a37c6d6d60c" exitCode=0
Nov 03 19:36:20.587191 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:20.587067    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal" event={"ID":"583fd7cf27a9a1aaee6d847adb49df9a","Type":"ContainerDied","Data":"32b1aa6939637b01a067414caad8da0bbeb9e2ddff9e9a2335313a37c6d6d60c"}
Nov 03 19:36:20.590476 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:20.590446    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal" event={"ID":"465e529db144367f617f146802de7125","Type":"ContainerStarted","Data":"cad7672a21a325988a408255f0c5367ae045c4b500f71d954a808eaecca83607"}
Nov 03 19:36:21.524684 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:21.524652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:21.525096 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:21.524766    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:21.615741 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:21.615701    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal" event={"ID":"583fd7cf27a9a1aaee6d847adb49df9a","Type":"ContainerStarted","Data":"cb7933fdab6412a6232106d1f9c7cc4486c5936dd2f2fedf10d6f71b7ff797b1"}
Nov 03 19:36:21.634843 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:21.634778    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal" podStartSLOduration=3.634757074 podStartE2EDuration="3.634757074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:20.659516053 +0000 UTC m=+4.676095096" watchObservedRunningTime="2026-11-03 19:36:21.634757074 +0000 UTC m=+5.651336160"
Nov 03 19:36:22.124697 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:22.123675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:22.124697 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:22.123737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:22.124697 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:22.123779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ndj\" (UniqueName: \"kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj\") pod \"network-check-target-vcw9t\" (UID: \"124e9f93-f05c-4eee-8054-ac67754550c3\") " pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:22.124697 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:22.123944    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:22.124697 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:22.123963    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:22.124697 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:22.123977    2579 projected.go:196] Error preparing data for projected volume kube-api-access-l4ndj for pod openshift-network-diagnostics/network-check-target-vcw9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:22.124697 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:22.124037    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj podName:124e9f93-f05c-4eee-8054-ac67754550c3 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:26.12401766 +0000 UTC m=+10.140596681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l4ndj" (UniqueName: "kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj") pod "network-check-target-vcw9t" (UID: "124e9f93-f05c-4eee-8054-ac67754550c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:22.124697 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:22.124456    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:22.124697 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:22.124508    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs podName:658b22ed-1324-4c81-b112-f051fdd3baa4 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:26.124491818 +0000 UTC m=+10.141070836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs") pod "network-metrics-daemon-bpk6k" (UID: "658b22ed-1324-4c81-b112-f051fdd3baa4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:22.124697 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:22.124579    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:22.124697 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:22.124619    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret podName:ae317c95-3690-4603-a062-652384d87f55 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:26.124606776 +0000 UTC m=+10.141185795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret") pod "global-pull-secret-syncer-hc967" (UID: "ae317c95-3690-4603-a062-652384d87f55") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:22.524470 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:22.524439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:22.524663 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:22.524572    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:22.524884 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:22.524868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:22.525175 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:22.524943    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:22.623949 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:22.623911    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc6cb168-a115-4918-a092-bc340accd868" containerID="3c9293b28b0ba902f2988b0e6b317b78e21e27c17515926d8e452ba7b18c7e29" exitCode=0
Nov 03 19:36:22.624098 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:22.624035    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt6hc" event={"ID":"fc6cb168-a115-4918-a092-bc340accd868","Type":"ContainerDied","Data":"3c9293b28b0ba902f2988b0e6b317b78e21e27c17515926d8e452ba7b18c7e29"}
Nov 03 19:36:22.650109 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:22.650057    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal" podStartSLOduration=4.65003814 podStartE2EDuration="4.65003814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:21.635523505 +0000 UTC m=+5.652102545" watchObservedRunningTime="2026-11-03 19:36:22.65003814 +0000 UTC m=+6.666617177"
Nov 03 19:36:23.523785 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:23.523752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:23.523945 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:23.523887    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:23.630955 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:23.630914    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6q6gg" event={"ID":"16b10ae2-69df-49f1-b0b3-77c98ee4cc3d","Type":"ContainerStarted","Data":"59947fcd775b57e6e33e7fbe2759162cbf81861fe4a28f90a6d0eeb334b99464"}
Nov 03 19:36:23.648900 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:23.648441    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6q6gg" podStartSLOduration=3.45718907 podStartE2EDuration="7.648428835s" totalImagesPullingTime="4.191239765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:23.647967633 +0000 UTC m=+7.664546670" watchObservedRunningTime="2026-11-03 19:36:23.648428835 +0000 UTC m=+7.665007871"
Nov 03 19:36:24.526633 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:24.526599    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:24.526796 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:24.526740    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:24.527259 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:24.527236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:24.527365 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:24.527340    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:24.638344 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:24.638265    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mmxqm" event={"ID":"ad627830-62cd-4f5b-bb63-76b5944b91ff","Type":"ContainerStarted","Data":"4ccddab80c6ffd367a2625b51da04bdd1a21321d9f7af4b5de30a72fe6952b28"}
Nov 03 19:36:24.655508 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:24.654773    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mmxqm" podStartSLOduration=3.4905354109999998 podStartE2EDuration="8.65475508s" totalImagesPullingTime="5.164219669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:24.654143032 +0000 UTC m=+8.670722069" watchObservedRunningTime="2026-11-03 19:36:24.65475508 +0000 UTC m=+8.671334120"
Nov 03 19:36:25.183561 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:25.183453    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:36:25.184722 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:25.184457    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:36:25.524283 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:25.524250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:25.524446 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:25.524390    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:25.648204 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:25.647696    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws" event={"ID":"4af39c8b-d06b-4569-bc27-ca478cc4852c","Type":"ContainerStarted","Data":"1a8fad3f1f30484b9281600af7b66becbd4b5bdc405a8d20473b0b1f1011c90b"}
Nov 03 19:36:26.158643 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:26.158605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:26.158828 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:26.158670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:26.158828 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:26.158705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ndj\" (UniqueName: \"kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj\") pod \"network-check-target-vcw9t\" (UID: \"124e9f93-f05c-4eee-8054-ac67754550c3\") " pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:26.158931 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:26.158861    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:26.158931 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:26.158880    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:26.158931 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:26.158893    2579 projected.go:196] Error preparing data for projected volume kube-api-access-l4ndj for pod openshift-network-diagnostics/network-check-target-vcw9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:26.159070 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:26.158953    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj podName:124e9f93-f05c-4eee-8054-ac67754550c3 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:34.158932379 +0000 UTC m=+18.175511409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l4ndj" (UniqueName: "kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj") pod "network-check-target-vcw9t" (UID: "124e9f93-f05c-4eee-8054-ac67754550c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:26.159070 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:26.159031    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:26.159070 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:26.159068    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs podName:658b22ed-1324-4c81-b112-f051fdd3baa4 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:34.159055922 +0000 UTC m=+18.175634949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs") pod "network-metrics-daemon-bpk6k" (UID: "658b22ed-1324-4c81-b112-f051fdd3baa4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:26.159238 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:26.159121    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:26.159238 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:26.159154    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret podName:ae317c95-3690-4603-a062-652384d87f55 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:34.159143399 +0000 UTC m=+18.175722429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret") pod "global-pull-secret-syncer-hc967" (UID: "ae317c95-3690-4603-a062-652384d87f55") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:26.524849 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:26.524814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:26.524945 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:26.524933    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:26.525074 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:26.525058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:26.525154 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:26.525137    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:26.655441 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:26.655377    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:36:26.659088 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:26.659057    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k9k97" event={"ID":"4dd8ccdd-1987-4200-8963-983dc4fd445d","Type":"ContainerStarted","Data":"3b658b4d95f47d41eb1ac9ef9be6d119404a7a8b31b2bb8f80a4efcc948ccdd7"}
Nov 03 19:36:26.674782 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:26.674338    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:36:26.674782 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:26.674666    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws" event={"ID":"4af39c8b-d06b-4569-bc27-ca478cc4852c","Type":"ContainerStarted","Data":"810759158f68fb6c7f87378d9deb3619cd6161a6c3fb0448c805872def0e862f"}
Nov 03 19:36:26.692635 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:26.692550    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-k9k97" podStartSLOduration=1.924020418 podStartE2EDuration="9.692529274s" totalImagesPullingTime="7.768508856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:26.691613687 +0000 UTC m=+10.708192725" watchObservedRunningTime="2026-11-03 19:36:26.692529274 +0000 UTC m=+10.709108311"
Nov 03 19:36:27.451087 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:27.450518    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:36:26.655396901Z","UUID":"715edee5-509f-4e66-ba76-b52f004a0755","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:36:27.451947 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:27.451926    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:36:27.452082 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:27.451969    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:36:27.524577 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:27.524545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:27.524767 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:27.524680    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:27.678995 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:27.678909    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc6cb168-a115-4918-a092-bc340accd868" containerID="3b129ad0cafac257d8eabfd88d25b13adc9d6c0492c7ccca6110bb5a3de8b3d3" exitCode=0
Nov 03 19:36:27.679458 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:27.679009    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt6hc" event={"ID":"fc6cb168-a115-4918-a092-bc340accd868","Type":"ContainerDied","Data":"3b129ad0cafac257d8eabfd88d25b13adc9d6c0492c7ccca6110bb5a3de8b3d3"}
Nov 03 19:36:27.681282 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:27.681256    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5ktfb" event={"ID":"e79d1e48-bb87-4a1b-9b88-95a1551e959c","Type":"ContainerStarted","Data":"02c61a2ffb9220a94db54d34f10451c9ce3af3fbf688a575f97b3d3078ba268e"}
Nov 03 19:36:27.721893 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:27.721840    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5ktfb" podStartSLOduration=2.345282118 podStartE2EDuration="10.72182103s" totalImagesPullingTime="8.376538912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:27.721503571 +0000 UTC m=+11.738082609" watchObservedRunningTime="2026-11-03 19:36:27.72182103 +0000 UTC m=+11.738400068"
Nov 03 19:36:28.525711 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:28.524626    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:28.525711 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:28.524680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:28.525711 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:28.524785    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:28.525711 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:28.524946    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:28.686225 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:28.686146    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws" event={"ID":"4af39c8b-d06b-4569-bc27-ca478cc4852c","Type":"ContainerStarted","Data":"44aa9df0cc9be547da9ffb631e8a8352d0efdf27eb020e3c9b9bcf71dd42e9a3"}
Nov 03 19:36:28.687987 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:28.687955    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5qqvk" event={"ID":"c43e3b1f-9e9c-4768-8c45-e779762180bd","Type":"ContainerStarted","Data":"db5e858d78c8e17eb500e06f997c86e3275b6748d5cd88476f9693e40d3c32dd"}
Nov 03 19:36:28.707439 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:28.707375    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws" podStartSLOduration=3.004383215 podStartE2EDuration="11.707359177s" totalImagesPullingTime="8.702975962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:28.706759139 +0000 UTC m=+12.723338173" watchObservedRunningTime="2026-11-03 19:36:28.707359177 +0000 UTC m=+12.723938212"
Nov 03 19:36:28.723922 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:28.723859    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5qqvk" podStartSLOduration=3.791416983 podStartE2EDuration="11.723837236s" totalImagesPullingTime="7.932420253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:28.72325122 +0000 UTC m=+12.739830248" watchObservedRunningTime="2026-11-03 19:36:28.723837236 +0000 UTC m=+12.740416273"
Nov 03 19:36:29.524273 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:29.524195    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:29.524429 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:29.524316    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:29.692745 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:29.692705    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc6cb168-a115-4918-a092-bc340accd868" containerID="38d0f483ba10836fe1a13f0a313e63e52485fbb1bcaa8018ed7bd4ac1f3de4cb" exitCode=0
Nov 03 19:36:29.693886 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:29.692761    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt6hc" event={"ID":"fc6cb168-a115-4918-a092-bc340accd868","Type":"ContainerDied","Data":"38d0f483ba10836fe1a13f0a313e63e52485fbb1bcaa8018ed7bd4ac1f3de4cb"}
Nov 03 19:36:29.695077 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:29.695044    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vbcxg" event={"ID":"2b6a3864-3172-40f0-9ab4-7327e2bf461c","Type":"ContainerStarted","Data":"9fd4c1373c40639f1f13ea9b5be23386977dbf0c302936771130947920e1f01e"}
Nov 03 19:36:29.737960 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:29.737913    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vbcxg" podStartSLOduration=3.227658135 podStartE2EDuration="13.737899407s" totalImagesPullingTime="10.510241272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:29.737457708 +0000 UTC m=+13.754036744" watchObservedRunningTime="2026-11-03 19:36:29.737899407 +0000 UTC m=+13.754478478"
Nov 03 19:36:30.524537 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:30.524509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:30.524682 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:30.524624    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:30.524718 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:30.524674    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:30.524782 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:30.524764    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:30.699585 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:30.699554    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt6hc" event={"ID":"fc6cb168-a115-4918-a092-bc340accd868","Type":"ContainerStarted","Data":"487e3f941ff588cc5856d84658a2607130f05c224a1341c3c57bd5c4347cb350"}
Nov 03 19:36:31.524163 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:31.523926    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:31.524333 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:31.524274    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:31.704934 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:31.704892    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc6cb168-a115-4918-a092-bc340accd868" containerID="487e3f941ff588cc5856d84658a2607130f05c224a1341c3c57bd5c4347cb350" exitCode=0
Nov 03 19:36:31.705321 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:31.704936    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt6hc" event={"ID":"fc6cb168-a115-4918-a092-bc340accd868","Type":"ContainerDied","Data":"487e3f941ff588cc5856d84658a2607130f05c224a1341c3c57bd5c4347cb350"}
Nov 03 19:36:32.364397 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:32.364361    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:36:32.364587 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:32.364507    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:36:32.365448 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:32.365212    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:36:32.524522 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:32.523940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:32.524522 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:32.523972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:32.524522 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:32.524067    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:32.524522 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:32.524453    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:33.524208 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:33.524160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:33.524786 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:33.524282    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:34.231249 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:34.231208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ndj\" (UniqueName: \"kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj\") pod \"network-check-target-vcw9t\" (UID: \"124e9f93-f05c-4eee-8054-ac67754550c3\") " pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:34.231389 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:34.231270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:34.231389 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:34.231293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:34.231389 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:34.231377    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:34.231389 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:34.231379    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:34.231551 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:34.231408    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:34.231551 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:34.231430    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs podName:658b22ed-1324-4c81-b112-f051fdd3baa4 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:50.231415074 +0000 UTC m=+34.247994101 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs") pod "network-metrics-daemon-bpk6k" (UID: "658b22ed-1324-4c81-b112-f051fdd3baa4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:34.231551 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:34.231432    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:34.231551 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:34.231447    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret podName:ae317c95-3690-4603-a062-652384d87f55 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:50.231440689 +0000 UTC m=+34.248019703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret") pod "global-pull-secret-syncer-hc967" (UID: "ae317c95-3690-4603-a062-652384d87f55") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:34.231551 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:34.231457    2579 projected.go:196] Error preparing data for projected volume kube-api-access-l4ndj for pod openshift-network-diagnostics/network-check-target-vcw9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:34.231551 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:34.231536    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj podName:124e9f93-f05c-4eee-8054-ac67754550c3 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:50.231518971 +0000 UTC m=+34.248098024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l4ndj" (UniqueName: "kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj") pod "network-check-target-vcw9t" (UID: "124e9f93-f05c-4eee-8054-ac67754550c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:34.524224 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:34.524202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:34.524583 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:34.524341    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:34.524583 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:34.524409    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:34.524583 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:34.524507    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:34.714162 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:34.713922    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc6cb168-a115-4918-a092-bc340accd868" containerID="97f4800c37ebec47064220fc1e5d031e506a6290aaa469d85471ef25c1011e94" exitCode=0
Nov 03 19:36:34.714293 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:34.713990    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt6hc" event={"ID":"fc6cb168-a115-4918-a092-bc340accd868","Type":"ContainerDied","Data":"97f4800c37ebec47064220fc1e5d031e506a6290aaa469d85471ef25c1011e94"}
Nov 03 19:36:34.722162 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:34.722133    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5" event={"ID":"2d6f9087-55e4-4e30-905a-3a1493336ded","Type":"ContainerStarted","Data":"115548212ec256b6b2d0413961603d282898752119fcf3efa7d4506fa292ffc7"}
Nov 03 19:36:34.722275 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:34.722230    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5" event={"ID":"2d6f9087-55e4-4e30-905a-3a1493336ded","Type":"ContainerStarted","Data":"e4a2cb8d5082db02d58b34ded413e914b33887131983cc67ced30f1293da71bb"}
Nov 03 19:36:34.722335 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:34.722291    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5" event={"ID":"2d6f9087-55e4-4e30-905a-3a1493336ded","Type":"ContainerStarted","Data":"583e4994ff6c183a4629f887ec1af04e61558533dfb4b1cb9a6cf8147884e004"}
Nov 03 19:36:34.722395 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:34.722341    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5" event={"ID":"2d6f9087-55e4-4e30-905a-3a1493336ded","Type":"ContainerStarted","Data":"2ee512ec4aaaf265dfeb5e6aeb2469aecdbb8a84ece1981da38b6489cb6da20a"}
Nov 03 19:36:34.722395 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:34.722391    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5" event={"ID":"2d6f9087-55e4-4e30-905a-3a1493336ded","Type":"ContainerStarted","Data":"7cb3bc5f358f24ec95ff0247e671bc7c23669408406dbf5d770401c0264ad858"}
Nov 03 19:36:34.722510 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:34.722439    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5" event={"ID":"2d6f9087-55e4-4e30-905a-3a1493336ded","Type":"ContainerStarted","Data":"d0c95f451e801bfe509671f1ed495f07a861d9c67383a8773f5f08751d87455b"}
Nov 03 19:36:35.524559 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:35.524528    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:35.524923 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:35.524630    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:35.726799 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:35.726763    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc6cb168-a115-4918-a092-bc340accd868" containerID="368f7c54dbb72eb4fe5110fa39387632214bd73a6943b68bbd1be7fc7405e32b" exitCode=0
Nov 03 19:36:35.726933 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:35.726818    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt6hc" event={"ID":"fc6cb168-a115-4918-a092-bc340accd868","Type":"ContainerDied","Data":"368f7c54dbb72eb4fe5110fa39387632214bd73a6943b68bbd1be7fc7405e32b"}
Nov 03 19:36:36.525071 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:36.525037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:36.525673 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:36.525212    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:36.525673 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:36.525241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:36.525673 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:36.525303    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:36.727551 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:36.727516    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:36.732177 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:36.732145    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bt6hc" event={"ID":"fc6cb168-a115-4918-a092-bc340accd868","Type":"ContainerStarted","Data":"6167111194d3fcd589c2e840f155aaf0f2a5b150d00c6974c5634a31241c7e84"}
Nov 03 19:36:36.758732 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:36.758693    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bt6hc" podStartSLOduration=7.510337832 podStartE2EDuration="19.758679339s" totalImagesPullingTime="12.248341507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:36.758339975 +0000 UTC m=+20.774919018" watchObservedRunningTime="2026-11-03 19:36:36.758679339 +0000 UTC m=+20.775258372"
Nov 03 19:36:36.904293 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:36.904255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:37.523600 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:37.523563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:37.523741 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:37.523667    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:37.737275 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:37.737236    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5" event={"ID":"2d6f9087-55e4-4e30-905a-3a1493336ded","Type":"ContainerStarted","Data":"2c8a6df376d20c5a678c00aa2a720c0dad618bec4566d3ff180e1a3c80038dbb"}
Nov 03 19:36:37.983764 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:37.983725    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:38.120667 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:38.120632    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:38.524302 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:38.524269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:38.524464 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:38.524400    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:38.524464 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:38.524438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:38.524614 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:38.524542    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:39.524272 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:39.524245    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:39.524850 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:39.524350    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:39.744120 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:39.744085    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5" event={"ID":"2d6f9087-55e4-4e30-905a-3a1493336ded","Type":"ContainerStarted","Data":"d9c3f64f21d81160d1fec51d2b02138d456145f3870e8433bf2f69b8ab779a25"}
Nov 03 19:36:39.744351 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:39.744332    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:39.744414 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:39.744400    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:39.759426 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:39.759402    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:39.778268 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:39.778182    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5" podStartSLOduration=8.079595377 podStartE2EDuration="23.778169967s" totalImagesPullingTime="15.69857459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:39.777448635 +0000 UTC m=+23.794027682" watchObservedRunningTime="2026-11-03 19:36:39.778169967 +0000 UTC m=+23.794749002"
Nov 03 19:36:40.017184 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.017141    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:40.192645 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.192550    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:40.326129 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.326095    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:40.524073 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.524048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:40.524210 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.524055    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:40.524210 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:40.524190    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:40.524309 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:40.524256    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:40.711828 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.711744    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:40.730716 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.730598    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vcw9t"]
Nov 03 19:36:40.731206 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.731184    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hc967"]
Nov 03 19:36:40.731314 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.731289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:40.731393 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:40.731374    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:40.732901 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.732878    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bpk6k"]
Nov 03 19:36:40.745995 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.745968    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:40.745995 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.745983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:40.746150 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:40.746055    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:40.746225 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:40.746206    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:40.746653 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.746630    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:40.761923 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.761898    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:36:40.846251 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:40.846214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:41.551043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:41.551010    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:41.771632 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:41.771601    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:41.906948 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:41.906865    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:42.524562 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:42.524531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:42.524724 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:42.524541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:42.524724 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:42.524660    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:42.524724 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:42.524541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:42.524724 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:42.524707    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:42.524929 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:42.524787    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:42.620265 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:42.620226    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:42.753164 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:42.753129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:42.876257 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:42.876164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:43.106731 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:43.106700    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:43.250680 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:43.250648    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:43.381888 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:43.381857    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:43.409891 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:43.409870    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5ktfb_e79d1e48-bb87-4a1b-9b88-95a1551e959c/dns-node-resolver/0.log"
Nov 03 19:36:44.113765 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:44.113725    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:44.241857 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:44.241811    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:44.369248 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:44.369207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:44.407701 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:44.407673    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mmxqm_ad627830-62cd-4f5b-bb63-76b5944b91ff/node-ca/0.log"
Nov 03 19:36:44.496519 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:44.496429    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:44.524630 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:44.524604    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:44.524770 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:44.524653    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:44.524770 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:44.524664    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:44.524770 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:44.524743    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:44.524905 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:44.524829    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:44.524943 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:44.524917    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:44.663907 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:44.663873    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:44.870589 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:44.870499    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:45.075727 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:45.075692    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:45.208998 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:45.208962    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:45.775755 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:45.775726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:46.275756 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:46.275719    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:46.406359 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:46.406324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:46.525140 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:46.525114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:46.525308 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:46.525193    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:46.525308 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:46.525228    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:46.525308 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:46.525271    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:46.525308 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:46.525292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:46.525514 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:46.525335    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:46.536715 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:46.536644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:46.679122 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:46.679093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:46.904190 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:46.904147    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:47.079743 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:47.079704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:47.680228 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:47.680184    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:47.809311 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:47.809277    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:47.964753 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:47.964716    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:48.090696 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:48.090658    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:48.275652 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:48.275567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:48.406158 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:48.406122    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:48.524543 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:48.524506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:48.524704 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:48.524506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:48.524704 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:48.524627    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:48.524766 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:48.524718    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:48.524766 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:48.524517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:48.524828 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:48.524797    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:48.538135 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:48.538064    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:48.677716 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:48.677684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:48.901760 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:48.901679    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:49.071051 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:49.071008    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:49.604835 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:49.604801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:49.722987 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:49.722953    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:50.243666 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:50.243626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:50.243666 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:50.243670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:50.244135 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:50.243693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ndj\" (UniqueName: \"kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj\") pod \"network-check-target-vcw9t\" (UID: \"124e9f93-f05c-4eee-8054-ac67754550c3\") " pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:50.244135 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:50.243787    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:50.244135 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:50.243850    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret podName:ae317c95-3690-4603-a062-652384d87f55 nodeName:}" failed. No retries permitted until 2026-11-03 19:37:22.243835778 +0000 UTC m=+66.260414804 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret") pod "global-pull-secret-syncer-hc967" (UID: "ae317c95-3690-4603-a062-652384d87f55") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:50.244135 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:50.243788    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:50.244135 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:50.243795    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:50.244135 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:50.243916    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs podName:658b22ed-1324-4c81-b112-f051fdd3baa4 nodeName:}" failed. No retries permitted until 2026-11-03 19:37:22.243904053 +0000 UTC m=+66.260483080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs") pod "network-metrics-daemon-bpk6k" (UID: "658b22ed-1324-4c81-b112-f051fdd3baa4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:50.244135 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:50.243930    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:50.244135 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:50.243945    2579 projected.go:196] Error preparing data for projected volume kube-api-access-l4ndj for pod openshift-network-diagnostics/network-check-target-vcw9t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:50.244135 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:50.243988    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj podName:124e9f93-f05c-4eee-8054-ac67754550c3 nodeName:}" failed. No retries permitted until 2026-11-03 19:37:22.243977316 +0000 UTC m=+66.260556339 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l4ndj" (UniqueName: "kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj") pod "network-check-target-vcw9t" (UID: "124e9f93-f05c-4eee-8054-ac67754550c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:50.272335 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:50.272297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:50.401268 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:50.401234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:50.524345 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:50.524269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:50.524584 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:50.524277    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:50.524584 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:50.524397    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:50.524584 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:50.524278    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:50.524584 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:50.524505    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:50.524584 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:50.524535    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:50.527708 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:50.527688    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:50.676908 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:50.676877    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:50.882902 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:50.882800    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:51.081748 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:51.081705    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:51.717825 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:51.717785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:52.284263 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:52.284226    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:52.477864 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:52.477826    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:52.524686 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:52.524662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:52.524826 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:52.524712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:52.524826 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:52.524723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:52.524826 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:52.524806    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:52.524964 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:52.524883    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:52.525009 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:52.524980    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:52.675000 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:52.674920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:52.869837 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:52.869803    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:53.078276 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:53.078247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:53.213018 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:53.212983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:53.345280 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:53.345183    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:53.471220 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:53.471185    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:53.610727 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:53.610632    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:53.879990 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:53.879914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:54.524602 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:54.524563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:54.524792 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:54.524563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:54.524792 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:54.524685    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:54.524792 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:54.524712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:54.524951 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:54.524792    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:54.524951 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:54.524853    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:55.708890 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:55.708857    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:55.834241 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:55.834207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:56.524882 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:56.524854    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:56.525044 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:56.524998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:56.525044 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:56.525010    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:56.525118 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:56.525104    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:56.525149 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:56.525113    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:56.525189 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:56.525170    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:36:58.524360 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:58.524333    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:36:58.524360 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:58.524350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:36:58.524792 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:36:58.524350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:36:58.524792 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:58.524430    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:36:58.524792 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:58.524519    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:36:58.524792 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:36:58.524587    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:37:00.523653 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:00.523621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:37:00.524057 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:00.523629    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:37:00.524057 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:37:00.523754    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpk6k" podUID="658b22ed-1324-4c81-b112-f051fdd3baa4"
Nov 03 19:37:00.524057 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:37:00.523820    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc967" podUID="ae317c95-3690-4603-a062-652384d87f55"
Nov 03 19:37:00.524057 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:00.523873    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:37:00.524057 ip-10-0-133-249 kubenswrapper[2579]: E1103 19:37:00.523928    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vcw9t" podUID="124e9f93-f05c-4eee-8054-ac67754550c3"
Nov 03 19:37:01.718230 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:01.718195    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:01.918237 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:01.918201    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:02.246052 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.246026    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeReady"
Nov 03 19:37:02.246217 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.246153    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 19:37:02.298542 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.298451    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kgm4l"]
Nov 03 19:37:02.300530 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.300505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.301277 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.301259    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5xsml"]
Nov 03 19:37:02.303000 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.302984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.304584 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.304563    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:37:02.304584 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.304572    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:37:02.304722 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.304575    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:37:02.304722 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.304680    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b2f85\""
Nov 03 19:37:02.312854 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.312783    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kgm4l"]
Nov 03 19:37:02.315533 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.315508    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5xsml"]
Nov 03 19:37:02.318006 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.317982    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pqv25"]
Nov 03 19:37:02.320508 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.320477    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pqv25"
Nov 03 19:37:02.329377 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.329348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/674ad509-03fd-4f35-bc70-3a3602457de9-config-volume\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.329459 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.329390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbk8q\" (UniqueName: \"kubernetes.io/projected/674ad509-03fd-4f35-bc70-3a3602457de9-kube-api-access-nbk8q\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.329459 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.329418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74569270-9489-4c9f-b51e-d819081baa5d-cert\") pod \"ingress-canary-pqv25\" (UID: \"74569270-9489-4c9f-b51e-d819081baa5d\") " pod="openshift-ingress-canary/ingress-canary-pqv25"
Nov 03 19:37:02.329459 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.329448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfpsh\" (UniqueName: \"kubernetes.io/projected/e50201a6-9ca6-4ff0-9e55-cc324745331e-kube-api-access-dfpsh\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.329587 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.329498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e50201a6-9ca6-4ff0-9e55-cc324745331e-crio-socket\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.329587 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.329516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/674ad509-03fd-4f35-bc70-3a3602457de9-metrics-tls\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.329587 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.329530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674ad509-03fd-4f35-bc70-3a3602457de9-tmp-dir\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.329587 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.329583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e50201a6-9ca6-4ff0-9e55-cc324745331e-data-volume\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.329706 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.329600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e50201a6-9ca6-4ff0-9e55-cc324745331e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.329706 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.329626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e50201a6-9ca6-4ff0-9e55-cc324745331e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.329706 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.329641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctw8s\" (UniqueName: \"kubernetes.io/projected/74569270-9489-4c9f-b51e-d819081baa5d-kube-api-access-ctw8s\") pod \"ingress-canary-pqv25\" (UID: \"74569270-9489-4c9f-b51e-d819081baa5d\") " pod="openshift-ingress-canary/ingress-canary-pqv25"
Nov 03 19:37:02.329936 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.329917    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pqv25"]
Nov 03 19:37:02.405134 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.405096    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:37:02.405134 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.405121    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:37:02.405134 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.405096    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:37:02.405456 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.405156    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:37:02.405456 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.405217    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgjzc\""
Nov 03 19:37:02.405456 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.405323    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:37:02.405456 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.405381    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:37:02.405666 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.405502    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qg9jj\""
Nov 03 19:37:02.430696 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.430665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74569270-9489-4c9f-b51e-d819081baa5d-cert\") pod \"ingress-canary-pqv25\" (UID: \"74569270-9489-4c9f-b51e-d819081baa5d\") " pod="openshift-ingress-canary/ingress-canary-pqv25"
Nov 03 19:37:02.430780 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.430704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfpsh\" (UniqueName: \"kubernetes.io/projected/e50201a6-9ca6-4ff0-9e55-cc324745331e-kube-api-access-dfpsh\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.430780 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.430729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e50201a6-9ca6-4ff0-9e55-cc324745331e-crio-socket\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.430878 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.430846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/674ad509-03fd-4f35-bc70-3a3602457de9-metrics-tls\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.430878 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.430872    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e50201a6-9ca6-4ff0-9e55-cc324745331e-crio-socket\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.430972 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.430879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674ad509-03fd-4f35-bc70-3a3602457de9-tmp-dir\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.430972 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.430956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e50201a6-9ca6-4ff0-9e55-cc324745331e-data-volume\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.431062 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.430980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e50201a6-9ca6-4ff0-9e55-cc324745331e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.431062 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.431014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e50201a6-9ca6-4ff0-9e55-cc324745331e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.431062 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.431038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctw8s\" (UniqueName: \"kubernetes.io/projected/74569270-9489-4c9f-b51e-d819081baa5d-kube-api-access-ctw8s\") pod \"ingress-canary-pqv25\" (UID: \"74569270-9489-4c9f-b51e-d819081baa5d\") " pod="openshift-ingress-canary/ingress-canary-pqv25"
Nov 03 19:37:02.431186 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.431070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/674ad509-03fd-4f35-bc70-3a3602457de9-config-volume\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.431186 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.431093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbk8q\" (UniqueName: \"kubernetes.io/projected/674ad509-03fd-4f35-bc70-3a3602457de9-kube-api-access-nbk8q\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.431290 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.431223    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674ad509-03fd-4f35-bc70-3a3602457de9-tmp-dir\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.431342 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.431305    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e50201a6-9ca6-4ff0-9e55-cc324745331e-data-volume\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.431698 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.431679    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/674ad509-03fd-4f35-bc70-3a3602457de9-config-volume\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.431777 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.431699    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e50201a6-9ca6-4ff0-9e55-cc324745331e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.434219 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.434200    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e50201a6-9ca6-4ff0-9e55-cc324745331e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.434293 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.434231    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/674ad509-03fd-4f35-bc70-3a3602457de9-metrics-tls\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.434335 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.434308    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74569270-9489-4c9f-b51e-d819081baa5d-cert\") pod \"ingress-canary-pqv25\" (UID: \"74569270-9489-4c9f-b51e-d819081baa5d\") " pod="openshift-ingress-canary/ingress-canary-pqv25"
Nov 03 19:37:02.442748 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.442709    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfpsh\" (UniqueName: \"kubernetes.io/projected/e50201a6-9ca6-4ff0-9e55-cc324745331e-kube-api-access-dfpsh\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.442884 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.442863    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbk8q\" (UniqueName: \"kubernetes.io/projected/674ad509-03fd-4f35-bc70-3a3602457de9-kube-api-access-nbk8q\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.443117 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.443095    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctw8s\" (UniqueName: \"kubernetes.io/projected/74569270-9489-4c9f-b51e-d819081baa5d-kube-api-access-ctw8s\") pod \"ingress-canary-pqv25\" (UID: \"74569270-9489-4c9f-b51e-d819081baa5d\") " pod="openshift-ingress-canary/ingress-canary-pqv25"
Nov 03 19:37:02.523743 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.523708    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:37:02.523913 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.523771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:37:02.523913 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.523778    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:37:02.528365 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.528338    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:37:02.528473 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.528415    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vlmgx\""
Nov 03 19:37:02.528473 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.528423    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:37:02.528590 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.528518    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx8mk\""
Nov 03 19:37:02.528590 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.528568    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5qm8x\""
Nov 03 19:37:02.529427 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.529415    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:37:02.529501 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.529436    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:37:02.614317 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.614237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:02.621039 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.621005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:02.629816 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.629788    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pqv25"
Nov 03 19:37:02.900185 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.900119    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5xsml"]
Nov 03 19:37:02.902926 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.902906    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kgm4l"]
Nov 03 19:37:02.903974 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:02.903852    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674ad509_03fd_4f35_bc70_3a3602457de9.slice/crio-d391ce127503ef4acd3720aeab598cc6b49a1b1c37bfbff9e968cb7e51f55432 WatchSource:0}: Error finding container d391ce127503ef4acd3720aeab598cc6b49a1b1c37bfbff9e968cb7e51f55432: Status 404 returned error can't find the container with id d391ce127503ef4acd3720aeab598cc6b49a1b1c37bfbff9e968cb7e51f55432
Nov 03 19:37:02.905819 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:02.905798    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50201a6_9ca6_4ff0_9e55_cc324745331e.slice/crio-a3c0675a6699e8ee3b795bacd4c367cb7109896eeb463faa73679150e9dc62e6 WatchSource:0}: Error finding container a3c0675a6699e8ee3b795bacd4c367cb7109896eeb463faa73679150e9dc62e6: Status 404 returned error can't find the container with id a3c0675a6699e8ee3b795bacd4c367cb7109896eeb463faa73679150e9dc62e6
Nov 03 19:37:02.909849 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.909829    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pqv25"]
Nov 03 19:37:02.913097 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:02.913080    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74569270_9489_4c9f_b51e_d819081baa5d.slice/crio-96328b245f60ae7829d009f60842f5c5c34b7a4dd849bad09594465546d487cf WatchSource:0}: Error finding container 96328b245f60ae7829d009f60842f5c5c34b7a4dd849bad09594465546d487cf: Status 404 returned error can't find the container with id 96328b245f60ae7829d009f60842f5c5c34b7a4dd849bad09594465546d487cf
Nov 03 19:37:02.958552 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.958528    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d4cc48b5f-9g4kc"]
Nov 03 19:37:02.961004 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.960985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:02.964869 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.964852    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 19:37:02.965174 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.965116    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 19:37:02.965174 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.965133    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 19:37:02.965174 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.965151    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 19:37:02.965174 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.965162    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 19:37:02.965174 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.965158    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 19:37:02.965907 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.965895    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 19:37:02.966062 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.966040    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-779gt\""
Nov 03 19:37:02.972609 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:02.972589    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d4cc48b5f-9g4kc"]
Nov 03 19:37:03.035741 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.035709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-config\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.035871 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.035842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-service-ca\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.035923 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.035871    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-serving-cert\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.035923 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.035891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs4rw\" (UniqueName: \"kubernetes.io/projected/51cdd330-78ed-418e-8bdb-1f080d7c2b11-kube-api-access-fs4rw\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.035923 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.035918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-oauth-config\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.036011 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.035933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-oauth-serving-cert\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.036043 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.036006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51cdd330-78ed-418e-8bdb-1f080d7c2b11-tmp\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.137062 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.137029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-service-ca\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.137062 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.137067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-serving-cert\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.137301 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.137086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4rw\" (UniqueName: \"kubernetes.io/projected/51cdd330-78ed-418e-8bdb-1f080d7c2b11-kube-api-access-fs4rw\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.137301 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.137111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-oauth-config\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.137301 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.137128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-oauth-serving-cert\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.137301 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.137155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51cdd330-78ed-418e-8bdb-1f080d7c2b11-tmp\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.137301 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.137215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-config\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.137700 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.137676    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51cdd330-78ed-418e-8bdb-1f080d7c2b11-tmp\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.137926 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.137910    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-oauth-serving-cert\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.138007 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.137976    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-service-ca\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.138007 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.137991    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-config\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.139579 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.139559    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-oauth-config\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.139579 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.139566    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-serving-cert\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.149025 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.149002    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4rw\" (UniqueName: \"kubernetes.io/projected/51cdd330-78ed-418e-8bdb-1f080d7c2b11-kube-api-access-fs4rw\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.270776 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.270740    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:03.549540 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.541117    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d4cc48b5f-9g4kc"]
Nov 03 19:37:03.553164 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:03.553135    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cdd330_78ed_418e_8bdb_1f080d7c2b11.slice/crio-8e9861ceb2125342bfad482387a49b5d772d504601db52983f25766fb5420c3a WatchSource:0}: Error finding container 8e9861ceb2125342bfad482387a49b5d772d504601db52983f25766fb5420c3a: Status 404 returned error can't find the container with id 8e9861ceb2125342bfad482387a49b5d772d504601db52983f25766fb5420c3a
Nov 03 19:37:03.793210 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.793124    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pqv25" event={"ID":"74569270-9489-4c9f-b51e-d819081baa5d","Type":"ContainerStarted","Data":"96328b245f60ae7829d009f60842f5c5c34b7a4dd849bad09594465546d487cf"}
Nov 03 19:37:03.794910 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.794862    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4cc48b5f-9g4kc" event={"ID":"51cdd330-78ed-418e-8bdb-1f080d7c2b11","Type":"ContainerStarted","Data":"8e9861ceb2125342bfad482387a49b5d772d504601db52983f25766fb5420c3a"}
Nov 03 19:37:03.797209 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.797183    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kgm4l" event={"ID":"e50201a6-9ca6-4ff0-9e55-cc324745331e","Type":"ContainerStarted","Data":"99886b645f4d332625e754bdd0c388624a2e0e857af0b7b68979aa48d695fba3"}
Nov 03 19:37:03.797333 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.797238    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kgm4l" event={"ID":"e50201a6-9ca6-4ff0-9e55-cc324745331e","Type":"ContainerStarted","Data":"c8102d0e1120d43340feb52529d6bab6153cd251d0d3765988ec9d2bc931ecee"}
Nov 03 19:37:03.797333 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.797271    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kgm4l" event={"ID":"e50201a6-9ca6-4ff0-9e55-cc324745331e","Type":"ContainerStarted","Data":"a3c0675a6699e8ee3b795bacd4c367cb7109896eeb463faa73679150e9dc62e6"}
Nov 03 19:37:03.798401 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:03.798375    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5xsml" event={"ID":"674ad509-03fd-4f35-bc70-3a3602457de9","Type":"ContainerStarted","Data":"d391ce127503ef4acd3720aeab598cc6b49a1b1c37bfbff9e968cb7e51f55432"}
Nov 03 19:37:04.490126 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:04.490052    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:04.630905 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:04.630791    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:04.766787 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:04.766665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:05.807005 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:05.806927    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kgm4l" event={"ID":"e50201a6-9ca6-4ff0-9e55-cc324745331e","Type":"ContainerStarted","Data":"967fadb53559010b0a8393fe17da4fc0c57f95f7246f647a040b3066136b6bb5"}
Nov 03 19:37:05.809171 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:05.809145    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5xsml" event={"ID":"674ad509-03fd-4f35-bc70-3a3602457de9","Type":"ContainerStarted","Data":"2c01d017258c25cfa802dd74f93dfd7fd503fc0a6e9d10b5b23847e0ccccec95"}
Nov 03 19:37:05.809309 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:05.809196    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5xsml" event={"ID":"674ad509-03fd-4f35-bc70-3a3602457de9","Type":"ContainerStarted","Data":"62fac7c815f26dbec92f9f47f766a108488d78faeff63352d8ba8041d858f18a"}
Nov 03 19:37:05.809705 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:05.809686    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:05.829408 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:05.829335    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kgm4l" podStartSLOduration=1.367803713 podStartE2EDuration="3.829320765s" totalImagesPullingTime="2.461517052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:37:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:05.827785518 +0000 UTC m=+49.844364555" watchObservedRunningTime="2026-11-03 19:37:05.829320765 +0000 UTC m=+49.845899803"
Nov 03 19:37:05.848916 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:05.848847    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-5xsml" podStartSLOduration=1.721362547 podStartE2EDuration="3.848827865s" totalImagesPullingTime="2.127465318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:05.847870066 +0000 UTC m=+49.864449102" watchObservedRunningTime="2026-11-03 19:37:05.848827865 +0000 UTC m=+49.865406902"
Nov 03 19:37:06.607644 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:06.607593    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:06.738149 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:06.738119    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:06.814969 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:06.814929    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pqv25" event={"ID":"74569270-9489-4c9f-b51e-d819081baa5d","Type":"ContainerStarted","Data":"bcd87598ba07ce2a8095baa820c99f15af018d9ab05040185e1495b419089038"}
Nov 03 19:37:06.816543 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:06.816512    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4cc48b5f-9g4kc" event={"ID":"51cdd330-78ed-418e-8bdb-1f080d7c2b11","Type":"ContainerStarted","Data":"360c65d6bd521802d5b2fa893983456cf9889aece84bf553d31dff0aee76a209"}
Nov 03 19:37:06.835779 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:06.835731    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pqv25" podStartSLOduration=1.820601173 podStartE2EDuration="4.835718538s" totalImagesPullingTime="3.015117365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:06.834887598 +0000 UTC m=+50.851466638" watchObservedRunningTime="2026-11-03 19:37:06.835718538 +0000 UTC m=+50.852297573"
Nov 03 19:37:06.855659 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:06.855608    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d4cc48b5f-9g4kc" podStartSLOduration=1.69920486 podStartE2EDuration="4.855593662s" totalImagesPullingTime="3.156388802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:06.855022608 +0000 UTC m=+50.871601634" watchObservedRunningTime="2026-11-03 19:37:06.855593662 +0000 UTC m=+50.872172689"
Nov 03 19:37:06.893030 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:06.892952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:08.658051 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.658013    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:08.840315 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.840281    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"]
Nov 03 19:37:08.842234 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.842219    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:08.849166 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.849148    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 19:37:08.850124 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.850105    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 19:37:08.850226 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.850213    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 19:37:08.850269 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.850252    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 19:37:08.850307 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.850270    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-w7r7w\""
Nov 03 19:37:08.850338 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.850327    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 19:37:08.856045 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.856023    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"]
Nov 03 19:37:08.881699 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.881666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d03e8db-60e2-4844-94c3-78f02bc19100-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:08.881837 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.881714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d03e8db-60e2-4844-94c3-78f02bc19100-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:08.881882 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.881856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d03e8db-60e2-4844-94c3-78f02bc19100-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:08.881916 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.881882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8wnj\" (UniqueName: \"kubernetes.io/projected/4d03e8db-60e2-4844-94c3-78f02bc19100-kube-api-access-m8wnj\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:08.908948 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.908851    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9wvlh"]
Nov 03 19:37:08.909288 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.909266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:08.910921 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.910906    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:08.950092 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.950063    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 19:37:08.950092 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.950077    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 19:37:08.950092 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.950071    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 19:37:08.950281 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.950147    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lhxcg\""
Nov 03 19:37:08.982694 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.982662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-wtmp\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:08.982845 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.982703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:08.982845 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.982794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d03e8db-60e2-4844-94c3-78f02bc19100-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:08.982845 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.982825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146c892e-d6ef-4062-94c4-6dae80875952-metrics-client-ca\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:08.982845 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.982844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-textfile\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:08.983027 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.982864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d03e8db-60e2-4844-94c3-78f02bc19100-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:08.983027 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.982882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp2b4\" (UniqueName: \"kubernetes.io/projected/146c892e-d6ef-4062-94c4-6dae80875952-kube-api-access-vp2b4\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:08.983027 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.982997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-tls\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:08.983176 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.983030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:08.983176 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.983138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8wnj\" (UniqueName: \"kubernetes.io/projected/4d03e8db-60e2-4844-94c3-78f02bc19100-kube-api-access-m8wnj\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:08.983176 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.983164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-sys\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:08.983327 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.983199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d03e8db-60e2-4844-94c3-78f02bc19100-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:08.983327 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.983304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-root\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:08.983819 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.983797    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d03e8db-60e2-4844-94c3-78f02bc19100-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:08.985415 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.985390    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d03e8db-60e2-4844-94c3-78f02bc19100-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:08.985518 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.985399    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d03e8db-60e2-4844-94c3-78f02bc19100-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:08.998731 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:08.998709    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8wnj\" (UniqueName: \"kubernetes.io/projected/4d03e8db-60e2-4844-94c3-78f02bc19100-kube-api-access-m8wnj\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:09.044054 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.044023    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:09.083831 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.083799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146c892e-d6ef-4062-94c4-6dae80875952-metrics-client-ca\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.083831 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.083834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-textfile\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.084033 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.083862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp2b4\" (UniqueName: \"kubernetes.io/projected/146c892e-d6ef-4062-94c4-6dae80875952-kube-api-access-vp2b4\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.084033 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.083969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-tls\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.084033 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.084001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.084196 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.084048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-sys\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.084196 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.084068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-root\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.084196 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.084095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-wtmp\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.084196 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.084128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.084196 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.084157    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-sys\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.084196 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.084178    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-root\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.084501 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.084242    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-textfile\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.084501 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.084298    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-wtmp\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.084705 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.084682    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146c892e-d6ef-4062-94c4-6dae80875952-metrics-client-ca\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.084705 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.084700    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.086224 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.086207    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-tls\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.086295 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.086259    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.095878 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.095860    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp2b4\" (UniqueName: \"kubernetes.io/projected/146c892e-d6ef-4062-94c4-6dae80875952-kube-api-access-vp2b4\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.153743 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.153709    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:09.166655 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.166625    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:09.222818 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.222785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:09.231178 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:09.231149    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146c892e_d6ef_4062_94c4_6dae80875952.slice/crio-2407f8e452cd56a6af3ec945e8304de673e2a28c30806b7d4aaf0b6f5efa66b6 WatchSource:0}: Error finding container 2407f8e452cd56a6af3ec945e8304de673e2a28c30806b7d4aaf0b6f5efa66b6: Status 404 returned error can't find the container with id 2407f8e452cd56a6af3ec945e8304de673e2a28c30806b7d4aaf0b6f5efa66b6
Nov 03 19:37:09.294025 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.293994    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:09.427542 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.427335    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"]
Nov 03 19:37:09.427625 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.427607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:09.429794 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:09.429768    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d03e8db_60e2_4844_94c3_78f02bc19100.slice/crio-6e0e44d033431d15a3b9e1eea482cc99809e57e9b9bfc4e8be326bb66ea0be06 WatchSource:0}: Error finding container 6e0e44d033431d15a3b9e1eea482cc99809e57e9b9bfc4e8be326bb66ea0be06: Status 404 returned error can't find the container with id 6e0e44d033431d15a3b9e1eea482cc99809e57e9b9bfc4e8be326bb66ea0be06
Nov 03 19:37:09.558733 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.558688    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:09.808871 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.808833    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:09.827545 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.827510    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx" event={"ID":"4d03e8db-60e2-4844-94c3-78f02bc19100","Type":"ContainerStarted","Data":"6d0da093d8bfd5c65015fd217bcc73b0dab7c2f603b4a071e26cf335d8b070f2"}
Nov 03 19:37:09.827702 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.827592    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx" event={"ID":"4d03e8db-60e2-4844-94c3-78f02bc19100","Type":"ContainerStarted","Data":"543dd701d20cd5c743fdad5fef9a5a8cbfa35f8166316f9d9895967a9529f6cb"}
Nov 03 19:37:09.827702 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.827620    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx" event={"ID":"4d03e8db-60e2-4844-94c3-78f02bc19100","Type":"ContainerStarted","Data":"6e0e44d033431d15a3b9e1eea482cc99809e57e9b9bfc4e8be326bb66ea0be06"}
Nov 03 19:37:09.828879 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.828857    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wvlh" event={"ID":"146c892e-d6ef-4062-94c4-6dae80875952","Type":"ContainerStarted","Data":"2407f8e452cd56a6af3ec945e8304de673e2a28c30806b7d4aaf0b6f5efa66b6"}
Nov 03 19:37:09.942645 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:09.942608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:10.081233 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.081145    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:10.247667 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.247630    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:10.345721 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.345692    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cdcc6bb85-48j6f"]
Nov 03 19:37:10.348279 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.348258    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.365535 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.365501    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cdcc6bb85-48j6f"]
Nov 03 19:37:10.365785 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.365761    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 19:37:10.395788 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.395751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-trusted-ca-bundle\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.395957 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.395806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-service-ca\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.396009 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.395959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-console-config\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.396107 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.396080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-serving-cert\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.396160 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.396124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8236a6c-f408-4996-906d-a73a8cb0069d-tmp\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.396211 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.396160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj9m7\" (UniqueName: \"kubernetes.io/projected/c8236a6c-f408-4996-906d-a73a8cb0069d-kube-api-access-sj9m7\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.396211 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.396203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-oauth-serving-cert\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.396314 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.396221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-oauth-config\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.497347 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.497312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-service-ca\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.497347 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.497352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-console-config\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.497580 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.497465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-serving-cert\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.497580 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.497504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8236a6c-f408-4996-906d-a73a8cb0069d-tmp\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.497580 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.497557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj9m7\" (UniqueName: \"kubernetes.io/projected/c8236a6c-f408-4996-906d-a73a8cb0069d-kube-api-access-sj9m7\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.497730 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.497603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-oauth-serving-cert\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.497730 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.497628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-oauth-config\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.497730 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.497673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-trusted-ca-bundle\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.497919 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.497897    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8236a6c-f408-4996-906d-a73a8cb0069d-tmp\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.498230 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.498200    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-service-ca\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.498357 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.498200    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-console-config\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.498357 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.498297    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-oauth-serving-cert\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.498587 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.498566    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-trusted-ca-bundle\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.500210 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.500190    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-oauth-config\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.500285 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.500260    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-serving-cert\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.509888 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.509867    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj9m7\" (UniqueName: \"kubernetes.io/projected/c8236a6c-f408-4996-906d-a73a8cb0069d-kube-api-access-sj9m7\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.659953 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.659866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:10.709120 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.709084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:10.833537 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.833476    2579 generic.go:356] "Generic (PLEG): container finished" podID="146c892e-d6ef-4062-94c4-6dae80875952" containerID="76755129c608a9f4e6ee81f4c869ab3d98e6254f762f4a6c06b4ea2f85447378" exitCode=0
Nov 03 19:37:10.833537 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.833519    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wvlh" event={"ID":"146c892e-d6ef-4062-94c4-6dae80875952","Type":"ContainerDied","Data":"76755129c608a9f4e6ee81f4c869ab3d98e6254f762f4a6c06b4ea2f85447378"}
Nov 03 19:37:10.835899 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.835873    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx" event={"ID":"4d03e8db-60e2-4844-94c3-78f02bc19100","Type":"ContainerStarted","Data":"46ff7c08270b886389e507d794d49ac8090d6c12ce8a7d1653c08872377766e5"}
Nov 03 19:37:10.863430 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.863401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:10.878080 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.878030    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx" podStartSLOduration=1.890352772 podStartE2EDuration="2.878016324s" totalImagesPullingTime="987.663552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:10.876942908 +0000 UTC m=+54.893521945" watchObservedRunningTime="2026-11-03 19:37:10.878016324 +0000 UTC m=+54.894595393"
Nov 03 19:37:10.928811 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:10.928787    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cdcc6bb85-48j6f"]
Nov 03 19:37:10.931758 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:10.931732    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8236a6c_f408_4996_906d_a73a8cb0069d.slice/crio-1eb57135de286a127233d228fa622464972983143f5729150279ebd5289647dc WatchSource:0}: Error finding container 1eb57135de286a127233d228fa622464972983143f5729150279ebd5289647dc: Status 404 returned error can't find the container with id 1eb57135de286a127233d228fa622464972983143f5729150279ebd5289647dc
Nov 03 19:37:11.840941 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:11.840909    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wvlh" event={"ID":"146c892e-d6ef-4062-94c4-6dae80875952","Type":"ContainerStarted","Data":"d212d2824a17b738fbaab006c63cf6e39f07ac6fe3080378c13262bac0a92c26"}
Nov 03 19:37:11.841411 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:11.840992    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wvlh" event={"ID":"146c892e-d6ef-4062-94c4-6dae80875952","Type":"ContainerStarted","Data":"2431308f7963a22ed0ef2e6cb2e4761dbc10f82d3719ba15c4d745f736e8d681"}
Nov 03 19:37:11.842444 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:11.842410    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdcc6bb85-48j6f" event={"ID":"c8236a6c-f408-4996-906d-a73a8cb0069d","Type":"ContainerStarted","Data":"dadcf4d7751bc8cea978607d5d6e9357e80705831cfd7918d5faeb18659924c3"}
Nov 03 19:37:11.842569 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:11.842460    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdcc6bb85-48j6f" event={"ID":"c8236a6c-f408-4996-906d-a73a8cb0069d","Type":"ContainerStarted","Data":"1eb57135de286a127233d228fa622464972983143f5729150279ebd5289647dc"}
Nov 03 19:37:11.863681 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:11.863650    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:11.864386 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:11.864348    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9wvlh" podStartSLOduration=2.719787222 podStartE2EDuration="3.864336755s" totalImagesPullingTime="1.066599675s" totalInitContainerRuntime="77.949858ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:11.862355336 +0000 UTC m=+55.878934375" watchObservedRunningTime="2026-11-03 19:37:11.864336755 +0000 UTC m=+55.880915790"
Nov 03 19:37:11.884895 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:11.884850    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6cdcc6bb85-48j6f" podStartSLOduration=1.88483782 podStartE2EDuration="1.88483782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:11.883901924 +0000 UTC m=+55.900480960" watchObservedRunningTime="2026-11-03 19:37:11.88483782 +0000 UTC m=+55.901416855"
Nov 03 19:37:11.991521 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:11.991469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:12.688608 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:12.688579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:12.761435 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:12.761405    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:12.821576 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:12.821536    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:13.257458 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.257429    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"]
Nov 03 19:37:13.259394 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.259379    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.271021 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.270994    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:13.271131 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.271055    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:13.275658 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.275640    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:13.301269 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.301242    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 19:37:13.302720 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.302519    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 19:37:13.302720 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.302566    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fqsxm\""
Nov 03 19:37:13.302720 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.302635    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 19:37:13.302720 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.302707    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4nue8o8t41bmn\""
Nov 03 19:37:13.303061 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.303038    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 19:37:13.303818 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.303802    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"]
Nov 03 19:37:13.321403 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.321371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-secret-metrics-server-tls\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.321670 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.321418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0edbce3-cc93-410d-b82e-27e13cb6412c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.321670 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.321556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-client-ca-bundle\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.321670 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.321580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plbb9\" (UniqueName: \"kubernetes.io/projected/a0edbce3-cc93-410d-b82e-27e13cb6412c-kube-api-access-plbb9\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.321670 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.321623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a0edbce3-cc93-410d-b82e-27e13cb6412c-audit-log\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.321860 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.321777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-secret-metrics-server-client-certs\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.321946 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.321928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a0edbce3-cc93-410d-b82e-27e13cb6412c-metrics-server-audit-profiles\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.422980 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.422938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a0edbce3-cc93-410d-b82e-27e13cb6412c-metrics-server-audit-profiles\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.422980 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.422983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-secret-metrics-server-tls\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.423215 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.423007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0edbce3-cc93-410d-b82e-27e13cb6412c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.423215 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.423060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-client-ca-bundle\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.423215 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.423086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plbb9\" (UniqueName: \"kubernetes.io/projected/a0edbce3-cc93-410d-b82e-27e13cb6412c-kube-api-access-plbb9\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.423215 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.423115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a0edbce3-cc93-410d-b82e-27e13cb6412c-audit-log\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.423215 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.423157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-secret-metrics-server-client-certs\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.423618 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.423585    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a0edbce3-cc93-410d-b82e-27e13cb6412c-audit-log\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.423895 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.423872    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0edbce3-cc93-410d-b82e-27e13cb6412c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.424063 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.424047    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a0edbce3-cc93-410d-b82e-27e13cb6412c-metrics-server-audit-profiles\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.425641 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.425617    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-secret-metrics-server-tls\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.425971 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.425955    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-secret-metrics-server-client-certs\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.426167 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.426150    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-client-ca-bundle\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.435532 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.435504    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plbb9\" (UniqueName: \"kubernetes.io/projected/a0edbce3-cc93-410d-b82e-27e13cb6412c-kube-api-access-plbb9\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.570046 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.569953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:13.616455 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.616421    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:13.746990 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.746956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:13.840626 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.840544    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"]
Nov 03 19:37:13.843426 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:13.843400    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0edbce3_cc93_410d_b82e_27e13cb6412c.slice/crio-b37a78448ca767d1b95c0194bee981aebc10b55e1732396d0a1b7bee4b2be2a9 WatchSource:0}: Error finding container b37a78448ca767d1b95c0194bee981aebc10b55e1732396d0a1b7bee4b2be2a9: Status 404 returned error can't find the container with id b37a78448ca767d1b95c0194bee981aebc10b55e1732396d0a1b7bee4b2be2a9
Nov 03 19:37:13.849535 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.849512    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr" event={"ID":"a0edbce3-cc93-410d-b82e-27e13cb6412c","Type":"ContainerStarted","Data":"b37a78448ca767d1b95c0194bee981aebc10b55e1732396d0a1b7bee4b2be2a9"}
Nov 03 19:37:13.854979 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.854960    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:13.879731 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:13.879703    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:14.080891 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.080858    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7755d4f754-nrkkn"]
Nov 03 19:37:14.083627 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.083611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.087709 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.087677    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 19:37:14.087709 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.087692    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 19:37:14.087897 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.087717    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 19:37:14.087897 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.087725    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mkqv6\""
Nov 03 19:37:14.087897 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.087810    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 19:37:14.087897 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.087885    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 19:37:14.091711 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.091660    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 19:37:14.099670 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.099643    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7755d4f754-nrkkn"]
Nov 03 19:37:14.130468 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.130436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-secret-telemeter-client\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.130468 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.130473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48pr9\" (UniqueName: \"kubernetes.io/projected/93b47942-ccbf-43cd-8722-c21038f906f3-kube-api-access-48pr9\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.130714 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.130509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.130714 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.130641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-federate-client-tls\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.130805 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.130741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.130838 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.130805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-serving-certs-ca-bundle\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.130838 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.130833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-metrics-client-ca\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.130909 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.130857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-telemeter-client-tls\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.231838 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.231797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-federate-client-tls\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.232038 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.231896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.232038 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.231928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-serving-certs-ca-bundle\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.232038 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.231956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-metrics-client-ca\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.232038 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.231989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-telemeter-client-tls\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.232038 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.232025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-secret-telemeter-client\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.232281 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.232057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48pr9\" (UniqueName: \"kubernetes.io/projected/93b47942-ccbf-43cd-8722-c21038f906f3-kube-api-access-48pr9\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.232281 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.232085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.232834 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.232804    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-metrics-client-ca\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.232958 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.232946    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-serving-certs-ca-bundle\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.233009 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.232952    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.234857 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.234837    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-federate-client-tls\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.234947 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.234915    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-secret-telemeter-client\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.234947 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.234927    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.235070 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.235008    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-telemeter-client-tls\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.244198 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.244181    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48pr9\" (UniqueName: \"kubernetes.io/projected/93b47942-ccbf-43cd-8722-c21038f906f3-kube-api-access-48pr9\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.395364 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.395272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:14.681919 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.681880    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7755d4f754-nrkkn"]
Nov 03 19:37:14.682983 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:14.682953    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b47942_ccbf_43cd_8722_c21038f906f3.slice/crio-a9136d471b1ca4ba4499fedde7fdb262c5c0e8dfc6b492c43a2581ac3c51c1c5 WatchSource:0}: Error finding container a9136d471b1ca4ba4499fedde7fdb262c5c0e8dfc6b492c43a2581ac3c51c1c5: Status 404 returned error can't find the container with id a9136d471b1ca4ba4499fedde7fdb262c5c0e8dfc6b492c43a2581ac3c51c1c5
Nov 03 19:37:14.854391 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:14.854348    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn" event={"ID":"93b47942-ccbf-43cd-8722-c21038f906f3","Type":"ContainerStarted","Data":"a9136d471b1ca4ba4499fedde7fdb262c5c0e8dfc6b492c43a2581ac3c51c1c5"}
Nov 03 19:37:15.176820 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.176781    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:37:15.179677 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.179655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.184892 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.184866    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 19:37:15.185007 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.184992    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 19:37:15.185440 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.185421    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 19:37:15.185561 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.185515    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1kr3opstr93m9\""
Nov 03 19:37:15.185621 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.185577    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 19:37:15.185684 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.185667    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 19:37:15.185739 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.185700    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 19:37:15.185739 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.185715    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 19:37:15.186527 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.186466    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zljmq\""
Nov 03 19:37:15.188027 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.188005    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 19:37:15.188148 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.188087    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 19:37:15.188300 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.188283    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 19:37:15.189628 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.189609    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 19:37:15.194623 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.194600    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 19:37:15.202024 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.201931    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:37:15.222209 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.222183    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:15.241863 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.241824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-web-config\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242047 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.241874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242047 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.241900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242047 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.241920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242047 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.241947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242047 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.241968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242047 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242047 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242047 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242355 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242355 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config-out\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242355 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6k67\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-kube-api-access-d6k67\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242355 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242355 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242355 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242355 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242355 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242355 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242697 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242358    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.242697 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.242372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.283648 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.283616    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 19:37:15.283648 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.283616    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 19:37:15.343528 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.343674 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.343674 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.343674 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.343844 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-web-config\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.343844 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.343844 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.343844 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.343844 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.343844 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.343844 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.343844 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.344223 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.344223 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.344223 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config-out\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.344223 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.343989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6k67\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-kube-api-access-d6k67\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.344223 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.344005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.344223 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.344041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.344223 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.344074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.344223 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.344102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.344618 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.344265    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.344877 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.344853    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.345826 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.345797    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.346594 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.346567    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.348440 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.348013    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-web-config\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.348440 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.348093    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.348440 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.348110    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.350740 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.349226    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.350740 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.349337    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.350740 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.349428    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.350740 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.349666    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.350740 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.349998    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.350740 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.350721    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.352007 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.351426    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.352007 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.351670    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config-out\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.354993 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.352630    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.354993 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.353889    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.356842 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.356822    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.357264 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.357244    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.358773 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.358752    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6k67\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-kube-api-access-d6k67\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.504640 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.504606    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:15.799910 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.799823    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:37:15.802905 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:15.802878    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6007b5bf_fe4e_47e8_b6d6_57f4b03768ec.slice/crio-41dfb349eab98c7c8e5cfe4b15803d9fb5393310c0f09b38bcf56d1d9f43273b WatchSource:0}: Error finding container 41dfb349eab98c7c8e5cfe4b15803d9fb5393310c0f09b38bcf56d1d9f43273b: Status 404 returned error can't find the container with id 41dfb349eab98c7c8e5cfe4b15803d9fb5393310c0f09b38bcf56d1d9f43273b
Nov 03 19:37:15.859234 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.859190    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr" event={"ID":"a0edbce3-cc93-410d-b82e-27e13cb6412c","Type":"ContainerStarted","Data":"2bdb9802e62d09dd9961538618b30277a91802abb78caf2d556d85e238d63bca"}
Nov 03 19:37:15.860616 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.860586    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerStarted","Data":"41dfb349eab98c7c8e5cfe4b15803d9fb5393310c0f09b38bcf56d1d9f43273b"}
Nov 03 19:37:15.880191 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:15.880134    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr" podStartSLOduration=1.42107032 podStartE2EDuration="2.880121221s" totalImagesPullingTime="1.459050901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:15.879248072 +0000 UTC m=+59.895827109" watchObservedRunningTime="2026-11-03 19:37:15.880121221 +0000 UTC m=+59.896700257"
Nov 03 19:37:16.611610 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:16.611523    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:16.734799 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:16.734720    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:16.821659 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:16.821629    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:16.867475 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:16.867386    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn" event={"ID":"93b47942-ccbf-43cd-8722-c21038f906f3","Type":"ContainerStarted","Data":"c77a99dc4d96dbf91fec553cb0d3e0a0052c935619797396e46207339f9330a9"}
Nov 03 19:37:17.215019 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:17.214975    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:17.351682 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:17.351654    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:17.872688 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:17.872652    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn" event={"ID":"93b47942-ccbf-43cd-8722-c21038f906f3","Type":"ContainerStarted","Data":"072a0ca5e967bdf70f45bc9efea8ca459a166b3464f4958b7a9cf11b18fb1308"}
Nov 03 19:37:17.873148 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:17.872708    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn" event={"ID":"93b47942-ccbf-43cd-8722-c21038f906f3","Type":"ContainerStarted","Data":"7611cc6d53644e1580d038c78585de7c2dd82a3aac5823761b3acb76f2d34dd8"}
Nov 03 19:37:17.874257 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:17.874233    2579 generic.go:356] "Generic (PLEG): container finished" podID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerID="208cb05cb678e62705b2663440c26a5ccc690ce877d907c6a205f699b8c76ae1" exitCode=0
Nov 03 19:37:17.874364 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:17.874280    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerDied","Data":"208cb05cb678e62705b2663440c26a5ccc690ce877d907c6a205f699b8c76ae1"}
Nov 03 19:37:17.898916 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:17.898863    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn" podStartSLOduration=1.395343387 podStartE2EDuration="3.898828886s" totalImagesPullingTime="2.503485499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:17.897586628 +0000 UTC m=+61.914165664" watchObservedRunningTime="2026-11-03 19:37:17.898828886 +0000 UTC m=+61.915407922"
Nov 03 19:37:18.027652 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:18.027616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:18.976716 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:18.976684    2579 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cdcc6bb85-48j6f"]
Nov 03 19:37:19.009435 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.009401    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67d4ffd4f9-rc92g"]
Nov 03 19:37:19.012329 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.012307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.021180 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.021154    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d4ffd4f9-rc92g"]
Nov 03 19:37:19.081596 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.081563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-trusted-ca-bundle\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.081789 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.081616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-oauth-serving-cert\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.081789 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.081724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-serving-cert\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.081789 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.081778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-service-ca\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.081923 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.081849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-console-config\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.081923 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.081877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7knzb\" (UniqueName: \"kubernetes.io/projected/99464aea-b429-4992-b4dd-ad48c7d99633-kube-api-access-7knzb\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.082000 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.081927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-oauth-config\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.082000 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.081955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99464aea-b429-4992-b4dd-ad48c7d99633-tmp\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.182874 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.182841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-trusted-ca-bundle\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.183057 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.182888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-oauth-serving-cert\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.183057 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.182910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-serving-cert\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.183057 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.182951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-service-ca\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.183057 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.182983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-console-config\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.183057 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.183007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7knzb\" (UniqueName: \"kubernetes.io/projected/99464aea-b429-4992-b4dd-ad48c7d99633-kube-api-access-7knzb\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.183057 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.183037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-oauth-config\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.183364 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.183062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99464aea-b429-4992-b4dd-ad48c7d99633-tmp\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.183687 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.183538    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99464aea-b429-4992-b4dd-ad48c7d99633-tmp\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.185044 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.184720    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-console-config\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.185044 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.184873    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-oauth-serving-cert\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.185044 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.185012    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-service-ca\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.185044 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.185038    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-trusted-ca-bundle\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.185857 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.185834    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-serving-cert\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.186730 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.186705    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-oauth-config\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.196026 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.196003    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7knzb\" (UniqueName: \"kubernetes.io/projected/99464aea-b429-4992-b4dd-ad48c7d99633-kube-api-access-7knzb\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.325552 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.325180    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:19.614628 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.614441    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d4ffd4f9-rc92g"]
Nov 03 19:37:19.617534 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:19.617501    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99464aea_b429_4992_b4dd_ad48c7d99633.slice/crio-0c2740c722e154dea4fa7c3ab8387754bcf4654e4dac8361ab8b0c0f5ea0effc WatchSource:0}: Error finding container 0c2740c722e154dea4fa7c3ab8387754bcf4654e4dac8361ab8b0c0f5ea0effc: Status 404 returned error can't find the container with id 0c2740c722e154dea4fa7c3ab8387754bcf4654e4dac8361ab8b0c0f5ea0effc
Nov 03 19:37:19.884547 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.884450    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d4ffd4f9-rc92g" event={"ID":"99464aea-b429-4992-b4dd-ad48c7d99633","Type":"ContainerStarted","Data":"51b36dea60f2863db390b45cb22a0675844a1ecd8adec21ab000a121eadc7097"}
Nov 03 19:37:19.884547 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.884524    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d4ffd4f9-rc92g" event={"ID":"99464aea-b429-4992-b4dd-ad48c7d99633","Type":"ContainerStarted","Data":"0c2740c722e154dea4fa7c3ab8387754bcf4654e4dac8361ab8b0c0f5ea0effc"}
Nov 03 19:37:19.909170 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:19.909113    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67d4ffd4f9-rc92g" podStartSLOduration=1.9090946880000002 podStartE2EDuration="1.909094688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:19.908598594 +0000 UTC m=+63.925177630" watchObservedRunningTime="2026-11-03 19:37:19.909094688 +0000 UTC m=+63.925673726"
Nov 03 19:37:20.056061 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:20.056027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:20.178686 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:20.178589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:20.660983 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:20.660939    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:21.469261 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.469222    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:21.589177 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.589147    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:21.771958 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.771881    2579 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d4ffd4f9-rc92g"]
Nov 03 19:37:21.799451 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.799414    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d6b976d69-rjgm5"]
Nov 03 19:37:21.802459 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.802443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:21.813516 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.813477    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6b976d69-rjgm5"]
Nov 03 19:37:21.893339 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.893307    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerStarted","Data":"0189b7f6d3ed356b8453d577364ebc70a887771aafae13ffed30c7e494a1d878"}
Nov 03 19:37:21.893534 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.893399    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerStarted","Data":"bbdd7e2ec527a85de1cad1e4981a42589b867a0ccc3d730013a818cca3d2adb4"}
Nov 03 19:37:21.912829 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.912799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-tmp\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:21.912829 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.912833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-oauth-serving-cert\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:21.913028 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.912862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-oauth-config\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:21.913028 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.912936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-trusted-ca-bundle\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:21.913099 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.913041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-service-ca\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:21.913099 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.913071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmfq\" (UniqueName: \"kubernetes.io/projected/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-kube-api-access-xrmfq\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:21.913177 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.913143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-config\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:21.913217 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:21.913181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-serving-cert\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.013887 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.013828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-service-ca\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.013887 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.013884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmfq\" (UniqueName: \"kubernetes.io/projected/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-kube-api-access-xrmfq\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.014135 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.013931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-config\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.014135 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.013954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-serving-cert\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.014135 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.013992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-tmp\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.014135 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.014010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-oauth-serving-cert\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.014135 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.014040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-oauth-config\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.014135 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.014054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-trusted-ca-bundle\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.014547 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.014524    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-service-ca\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.014860 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.014831    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-trusted-ca-bundle\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.015229 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.015188    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-tmp\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.015556 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.015457    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-oauth-serving-cert\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.015802 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.015768    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-config\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.017780 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.017758    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-oauth-config\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.017880 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.017783    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-serving-cert\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.027228 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.027171    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmfq\" (UniqueName: \"kubernetes.io/projected/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-kube-api-access-xrmfq\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.113445 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.113405    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:22.316452 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.316360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ndj\" (UniqueName: \"kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj\") pod \"network-check-target-vcw9t\" (UID: \"124e9f93-f05c-4eee-8054-ac67754550c3\") " pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:37:22.316452 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.316436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:37:22.316667 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.316471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:37:22.321350 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.320105    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:37:22.321350 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.320239    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:37:22.321350 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.320293    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:37:22.329596 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.329564    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:37:22.329728 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.329660    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:37:22.329920 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.329900    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:37:22.339808 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.339783    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4ndj\" (UniqueName: \"kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj\") pod \"network-check-target-vcw9t\" (UID: \"124e9f93-f05c-4eee-8054-ac67754550c3\") " pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:37:22.339963 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.339931    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx8mk\""
Nov 03 19:37:22.343859 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.343834    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5qm8x\""
Nov 03 19:37:22.347324 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.347286    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:37:22.347583 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.347567    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vlmgx\""
Nov 03 19:37:22.351111 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.351090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:37:22.354993 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.354972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:37:22.405325 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.405198    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6b976d69-rjgm5"]
Nov 03 19:37:22.415957 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:22.408239    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce3bacc_fa93_4880_983e_ac1b2ae7e488.slice/crio-a85fa15687989263b0d138222aa85fea42b528f5b63b41cf9c82060381afeeb8 WatchSource:0}: Error finding container a85fa15687989263b0d138222aa85fea42b528f5b63b41cf9c82060381afeeb8: Status 404 returned error can't find the container with id a85fa15687989263b0d138222aa85fea42b528f5b63b41cf9c82060381afeeb8
Nov 03 19:37:22.634337 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.634261    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hc967"]
Nov 03 19:37:22.637572 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:22.637547    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae317c95_3690_4603_a062_652384d87f55.slice/crio-799e49e8455b73a0798a741ae56920d0b8edac42e1b47a575aa8cbe1bf30263d WatchSource:0}: Error finding container 799e49e8455b73a0798a741ae56920d0b8edac42e1b47a575aa8cbe1bf30263d: Status 404 returned error can't find the container with id 799e49e8455b73a0798a741ae56920d0b8edac42e1b47a575aa8cbe1bf30263d
Nov 03 19:37:22.637847 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.637826    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vcw9t"]
Nov 03 19:37:22.640052 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:22.640027    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124e9f93_f05c_4eee_8054_ac67754550c3.slice/crio-cdcb2de077db9dfd6db8c0ca41da578a27118ffae4f6f3e03a924e235d93f950 WatchSource:0}: Error finding container cdcb2de077db9dfd6db8c0ca41da578a27118ffae4f6f3e03a924e235d93f950: Status 404 returned error can't find the container with id cdcb2de077db9dfd6db8c0ca41da578a27118ffae4f6f3e03a924e235d93f950
Nov 03 19:37:22.647759 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.647739    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bpk6k"]
Nov 03 19:37:22.650048 ip-10-0-133-249 kubenswrapper[2579]: W1103 19:37:22.650022    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658b22ed_1324_4c81_b112_f051fdd3baa4.slice/crio-910af620224e6213f2d144032a30182959fcbcef6f0ba1a8847a8801c630a42e WatchSource:0}: Error finding container 910af620224e6213f2d144032a30182959fcbcef6f0ba1a8847a8801c630a42e: Status 404 returned error can't find the container with id 910af620224e6213f2d144032a30182959fcbcef6f0ba1a8847a8801c630a42e
Nov 03 19:37:22.651463 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.651438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:22.785191 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.785156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:22.898446 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.898325    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hc967" event={"ID":"ae317c95-3690-4603-a062-652384d87f55","Type":"ContainerStarted","Data":"799e49e8455b73a0798a741ae56920d0b8edac42e1b47a575aa8cbe1bf30263d"}
Nov 03 19:37:22.899769 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.899742    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bpk6k" event={"ID":"658b22ed-1324-4c81-b112-f051fdd3baa4","Type":"ContainerStarted","Data":"910af620224e6213f2d144032a30182959fcbcef6f0ba1a8847a8801c630a42e"}
Nov 03 19:37:22.901028 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.900999    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vcw9t" event={"ID":"124e9f93-f05c-4eee-8054-ac67754550c3","Type":"ContainerStarted","Data":"cdcb2de077db9dfd6db8c0ca41da578a27118ffae4f6f3e03a924e235d93f950"}
Nov 03 19:37:22.902586 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.902558    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6b976d69-rjgm5" event={"ID":"1ce3bacc-fa93-4880-983e-ac1b2ae7e488","Type":"ContainerStarted","Data":"809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b"}
Nov 03 19:37:22.902691 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.902600    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6b976d69-rjgm5" event={"ID":"1ce3bacc-fa93-4880-983e-ac1b2ae7e488","Type":"ContainerStarted","Data":"a85fa15687989263b0d138222aa85fea42b528f5b63b41cf9c82060381afeeb8"}
Nov 03 19:37:22.924322 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:22.924262    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7d6b976d69-rjgm5" podStartSLOduration=1.924246033 podStartE2EDuration="1.924246033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:22.922987309 +0000 UTC m=+66.939566360" watchObservedRunningTime="2026-11-03 19:37:22.924246033 +0000 UTC m=+66.940825062"
Nov 03 19:37:23.915433 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:23.915405    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerStarted","Data":"ec9eb9115176013e15a5c1fb9a8470723d3e4c3f26c228e2984e46487473debd"}
Nov 03 19:37:23.915842 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:23.915499    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerStarted","Data":"d8955bb3913556a7dcf78ded1a893ac224703af86c2c9254f99ac6d552bc4b08"}
Nov 03 19:37:24.445225 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:24.445089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:24.927752 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:24.927553    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerStarted","Data":"bf51b150c27764b3bdee088353bc5a57232cd08bd66985b19dce786935f1c1ea"}
Nov 03 19:37:24.927752 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:24.927692    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerStarted","Data":"c7400fea3ba057666ea68664c8c083a9f1531956c7284e97fb18a0ee40503e5b"}
Nov 03 19:37:24.929905 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:24.929798    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bpk6k" event={"ID":"658b22ed-1324-4c81-b112-f051fdd3baa4","Type":"ContainerStarted","Data":"62dfae4cdc67af94d9a05037923bdaaebe628fb4d6232c1b47ad35d7f81fdd67"}
Nov 03 19:37:24.929905 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:24.929852    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bpk6k" event={"ID":"658b22ed-1324-4c81-b112-f051fdd3baa4","Type":"ContainerStarted","Data":"2817d856b0d1f2ae6118e148211496b06789a63c7ee246db2a7a2c138a6da439"}
Nov 03 19:37:24.974989 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:24.974149    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.982573684 podStartE2EDuration="9.974130601s" totalImagesPullingTime="6.97529194s" totalInitContainerRuntime="16.264977ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:24.971264462 +0000 UTC m=+68.987843500" watchObservedRunningTime="2026-11-03 19:37:24.974130601 +0000 UTC m=+68.990709637"
Nov 03 19:37:25.249048 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:25.249007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:25.383766 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:25.383729    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:25.505179 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:25.505091    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:25.711220 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:25.711160    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:25.846337 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:25.846262    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:26.045229 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:26.045160    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:26.180322 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:26.180198    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:26.941180 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:26.941137    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vcw9t" event={"ID":"124e9f93-f05c-4eee-8054-ac67754550c3","Type":"ContainerStarted","Data":"63ed4ead9b48a1ef206f7529fdef494af877714baab8825c163a4cb5b9360c11"}
Nov 03 19:37:26.941368 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:26.941228    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:37:26.960922 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:26.960873    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bpk6k" podStartSLOduration=68.561782249 podStartE2EDuration="1m9.96085777s" totalImagesPullingTime="1.399075521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:24.994700803 +0000 UTC m=+69.011279839" watchObservedRunningTime="2026-11-03 19:37:26.96085777 +0000 UTC m=+70.977436806"
Nov 03 19:37:26.961118 ip-10-0-133-249 kubenswrapper[2579]: I1103 19:37:26.960978    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vcw9t" podStartSLOduration=66.404088214 podStartE2EDuration="1m9.960972041s" totalImagesPullingTime="3.556883827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:26.959274499 +0000 UTC m=+70.975853532" watchObservedRunningTime="2026-11-03 19:37:26.960972041 +0000 UTC m=+70.977551080"
Nov 03 19:37:27.774621 ip-10-0-133-249 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 19:37:27.783918 ip-10-0-133-249 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 19:37:27.784133 ip-10-0-133-249 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 19:37:27.784193 ip-10-0-133-249 systemd[1]: kubelet.service: Consumed 4.387s CPU time.
Nov 03 19:37:27.799081 ip-10-0-133-249 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:37:27.878743 ip-10-0-133-249 kubenswrapper[5659]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:37:27.878743 ip-10-0-133-249 kubenswrapper[5659]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:37:27.878743 ip-10-0-133-249 kubenswrapper[5659]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:37:27.878743 ip-10-0-133-249 kubenswrapper[5659]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:37:27.880979 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.880870    5659 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:27.880979 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.880981    5659 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.880988    5659 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.880992    5659 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.880997    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881002    5659 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881007    5659 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881011    5659 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881015    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881019    5659 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881023    5659 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881027    5659 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881031    5659 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881035    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881039    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881044    5659 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881050    5659 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881054    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881058    5659 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881062    5659 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:27.881071 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881066    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881070    5659 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881075    5659 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881080    5659 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881085    5659 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881089    5659 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881093    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881097    5659 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881101    5659 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881106    5659 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881110    5659 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881114    5659 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881118    5659 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881123    5659 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881126    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881130    5659 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881136    5659 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881143    5659 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881147    5659 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881152    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:27.881606 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881158    5659 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881162    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881168    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881173    5659 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881178    5659 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881183    5659 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881188    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881193    5659 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881197    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881201    5659 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881205    5659 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881209    5659 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881213    5659 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881218    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881222    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881226    5659 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881229    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881233    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881237    5659 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881241    5659 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:27.882098 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881246    5659 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881250    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881265    5659 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881269    5659 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881272    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881275    5659 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881278    5659 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881280    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881283    5659 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881287    5659 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881292    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881295    5659 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881298    5659 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881301    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881303    5659 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881307    5659 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881310    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881312    5659 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881316    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:27.882619 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881319    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881321    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881325    5659 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881327    5659 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881330    5659 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881332    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881335    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881338    5659 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881340    5659 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881343    5659 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881346    5659 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881348    5659 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881351    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881354    5659 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881357    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881359    5659 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881362    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881364    5659 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881367    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881370    5659 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:27.883101 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881372    5659 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881375    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881377    5659 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881380    5659 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881382    5659 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881385    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.881389    5659 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882179    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882187    5659 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882191    5659 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882194    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882196    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882200    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882203    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882206    5659 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882209    5659 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882212    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882215    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882217    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882220    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:27.883722 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882223    5659 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882226    5659 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882228    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882231    5659 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882236    5659 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882239    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882242    5659 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882245    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882248    5659 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882250    5659 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882253    5659 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882256    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882258    5659 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882261    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882263    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882266    5659 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882269    5659 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882271    5659 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882274    5659 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:27.884228 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882277    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882281    5659 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882283    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882286    5659 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882289    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882292    5659 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882295    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882298    5659 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882301    5659 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882303    5659 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882306    5659 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882308    5659 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882311    5659 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882313    5659 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882316    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882318    5659 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882321    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882323    5659 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882326    5659 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882328    5659 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:27.884754 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882331    5659 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882334    5659 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882336    5659 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882339    5659 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882342    5659 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882345    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882347    5659 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882350    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882352    5659 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882355    5659 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882357    5659 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882360    5659 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882362    5659 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882366    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882369    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882372    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882375    5659 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882378    5659 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882381    5659 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882384    5659 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:27.885254 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882386    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882389    5659 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882392    5659 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882394    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882397    5659 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882399    5659 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882402    5659 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882404    5659 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882407    5659 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882410    5659 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882412    5659 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882415    5659 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882417    5659 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882420    5659 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882422    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882424    5659 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882427    5659 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882430    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882432    5659 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882434    5659 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:27.885809 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882437    5659 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882474    5659 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882477    5659 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882502    5659 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882507    5659 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882511    5659 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882514    5659 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882518    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882521    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882524    5659 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882526    5659 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882529    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882533    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.882535    5659 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883079    5659 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883084    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883088    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883090    5659 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883093    5659 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883096    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:27.886309 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883099    5659 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883102    5659 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883104    5659 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883107    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883125    5659 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883129    5659 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883131    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883134    5659 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883137    5659 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883139    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883142    5659 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883145    5659 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883148    5659 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883151    5659 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883154    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883157    5659 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883159    5659 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883162    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883165    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883168    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:27.886822 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883170    5659 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883173    5659 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883175    5659 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883178    5659 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883182    5659 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883184    5659 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883187    5659 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883189    5659 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883192    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883195    5659 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883198    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883201    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883203    5659 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883205    5659 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883208    5659 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883218    5659 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883222    5659 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883229    5659 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883232    5659 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883235    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:27.887341 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883237    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883240    5659 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883243    5659 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883245    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883248    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883250    5659 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883253    5659 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883256    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883259    5659 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883263    5659 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883267    5659 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883271    5659 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883275    5659 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883278    5659 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883282    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883285    5659 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883288    5659 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883292    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:27.887850 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883295    5659 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883297    5659 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883300    5659 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883304    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883309    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883312    5659 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883316    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883320    5659 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883324    5659 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883327    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883330    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883334    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883337    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883339    5659 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883342    5659 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883344    5659 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883347    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883350    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883353    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:27.888298 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883355    5659 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883358    5659 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883361    5659 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883363    5659 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883366    5659 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883369    5659 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883372    5659 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883375    5659 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883378    5659 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883381    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883385    5659 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883389    5659 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883393    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883398    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883402    5659 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883404    5659 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883407    5659 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883410    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883412    5659 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883414    5659 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:27.888787 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883417    5659 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883420    5659 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.883422    5659 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883549    5659 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883563    5659 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883571    5659 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883575    5659 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883580    5659 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883583    5659 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883587    5659 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883592    5659 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883595    5659 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883598    5659 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883602    5659 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883605    5659 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883609    5659 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883612    5659 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883615    5659 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883618    5659 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883621    5659 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883624    5659 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883629    5659 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883632    5659 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:37:27.889273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883635    5659 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883639    5659 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883642    5659 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883647    5659 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883650    5659 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883654    5659 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883657    5659 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883660    5659 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883663    5659 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883666    5659 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883669    5659 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883672    5659 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883677    5659 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883682    5659 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883685    5659 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883688    5659 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883691    5659 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883694    5659 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883700    5659 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883703    5659 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883706    5659 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883709    5659 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883712    5659 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883717    5659 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:37:27.889844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883720    5659 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883723    5659 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883727    5659 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883730    5659 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883733    5659 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883736    5659 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883739    5659 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883742    5659 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883745    5659 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883748    5659 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883752    5659 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883755    5659 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883760    5659 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883763    5659 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883766    5659 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883769    5659 flags.go:64] FLAG: --help="false"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883772    5659 flags.go:64] FLAG: --hostname-override="ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883776    5659 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883779    5659 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883782    5659 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883786    5659 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883790    5659 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883793    5659 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:37:27.890462 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883798    5659 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883801    5659 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883804    5659 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883807    5659 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883810    5659 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883813    5659 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883816    5659 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883819    5659 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883822    5659 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883825    5659 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883828    5659 flags.go:64] FLAG: --lock-file=""
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883831    5659 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883834    5659 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883837    5659 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883843    5659 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883846    5659 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883850    5659 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883852    5659 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883855    5659 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883859    5659 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883862    5659 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883865    5659 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883871    5659 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883875    5659 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883878    5659 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:37:27.891074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883882    5659 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883885    5659 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883888    5659 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883891    5659 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883894    5659 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883898    5659 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883901    5659 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883910    5659 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883913    5659 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883916    5659 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883919    5659 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883922    5659 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883925    5659 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883928    5659 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883932    5659 flags.go:64] FLAG: --port="10250"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883935    5659 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883938    5659 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0680dc56ed20d9c03"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883941    5659 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883944    5659 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883947    5659 flags.go:64] FLAG: --register-node="true"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883950    5659 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883954    5659 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883957    5659 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883960    5659 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883963    5659 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883967    5659 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:37:27.892317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883970    5659 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883974    5659 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883976    5659 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883979    5659 flags.go:64] FLAG: --runonce="false"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883983    5659 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883987    5659 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883990    5659 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883993    5659 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883996    5659 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.883999    5659 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884002    5659 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884007    5659 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884010    5659 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884014    5659 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884017    5659 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884020    5659 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884023    5659 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884026    5659 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884029    5659 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884035    5659 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884038    5659 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884041    5659 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884046    5659 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884048    5659 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884051    5659 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:37:27.893004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884054    5659 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884057    5659 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884061    5659 flags.go:64] FLAG: --v="2"
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884065    5659 flags.go:64] FLAG: --version="false"
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884069    5659 flags.go:64] FLAG: --vmodule=""
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884073    5659 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884077    5659 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884221    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884225    5659 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884228    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884231    5659 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884234    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884238    5659 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884241    5659 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884244    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884246    5659 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884249    5659 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884251    5659 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884255    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884258    5659 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884262    5659 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:27.893690 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884264    5659 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884267    5659 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884270    5659 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884273    5659 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884275    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884278    5659 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884282    5659 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884286    5659 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884289    5659 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884291    5659 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884294    5659 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884320    5659 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884323    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884326    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884329    5659 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884331    5659 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884334    5659 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884336    5659 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884339    5659 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884341    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:27.894259 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884344    5659 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884346    5659 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884349    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884352    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884355    5659 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884358    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884361    5659 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884363    5659 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884366    5659 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884370    5659 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884373    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884375    5659 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884379    5659 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884384    5659 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884387    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884390    5659 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884392    5659 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884395    5659 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884398    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:27.894929 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884400    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884403    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884405    5659 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884408    5659 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884410    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884413    5659 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884416    5659 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884418    5659 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884421    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884424    5659 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884426    5659 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884429    5659 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884432    5659 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884434    5659 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884437    5659 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884440    5659 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884442    5659 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884445    5659 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884448    5659 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884451    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:27.895982 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884453    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884455    5659 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884459    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884462    5659 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884465    5659 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884469    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884472    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884474    5659 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884477    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884494    5659 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884497    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884499    5659 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884502    5659 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884504    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884507    5659 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884510    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884512    5659 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884515    5659 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884518    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:27.896620 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884520    5659 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884523    5659 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884525    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884528    5659 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884530    5659 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884533    5659 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884536    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884538    5659 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884541    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884543    5659 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884546    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884548    5659 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884559    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884562    5659 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.884591    5659 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884635    5659 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:27.897118 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884641    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884644    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884648    5659 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884651    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884654    5659 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884657    5659 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884659    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884662    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884665    5659 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884667    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884670    5659 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884672    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884675    5659 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884677    5659 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884680    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884682    5659 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884685    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884687    5659 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884690    5659 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:27.897637 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884693    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884695    5659 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884698    5659 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884700    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884703    5659 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884705    5659 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884708    5659 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884711    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884713    5659 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884716    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884719    5659 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884722    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884724    5659 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884729    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884732    5659 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884735    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884738    5659 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884740    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884743    5659 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884745    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:27.898141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884748    5659 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884751    5659 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884753    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884756    5659 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884759    5659 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884761    5659 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884764    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884767    5659 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884770    5659 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884772    5659 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884775    5659 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884777    5659 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884780    5659 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884782    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884785    5659 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884788    5659 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884792    5659 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884795    5659 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884797    5659 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884800    5659 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:27.898840 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884803    5659 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884806    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884809    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884812    5659 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884815    5659 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884819    5659 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884822    5659 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884826    5659 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884828    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884831    5659 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884833    5659 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884836    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884839    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884841    5659 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884844    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884847    5659 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884849    5659 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884851    5659 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884854    5659 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884857    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:27.899701 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884859    5659 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884862    5659 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884866    5659 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884869    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884872    5659 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884875    5659 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884877    5659 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884880    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884882    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884885    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884888    5659 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884890    5659 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884893    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884895    5659 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884898    5659 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884900    5659 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884903    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884908    5659 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884911    5659 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:27.900631 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884915    5659 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884918    5659 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884920    5659 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884923    5659 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884925    5659 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884928    5659 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.884930    5659 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.888890    5659 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.888907    5659 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.888927    5659 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.888932    5659 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.888983    5659 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.888988    5659 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.888993    5659 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.888996    5659 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.888999    5659 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889002    5659 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889005    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889008    5659 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:27.901412 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889010    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889013    5659 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889016    5659 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889018    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889021    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889024    5659 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889026    5659 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889029    5659 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889032    5659 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889034    5659 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889037    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889040    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889043    5659 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889046    5659 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889048    5659 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889051    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889054    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889057    5659 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889059    5659 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:27.902260 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889062    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889065    5659 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889068    5659 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889071    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889074    5659 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889076    5659 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889079    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889082    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889084    5659 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889086    5659 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889089    5659 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889092    5659 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889094    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889098    5659 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889103    5659 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889106    5659 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889109    5659 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889112    5659 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889115    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:27.903050 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889118    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889120    5659 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889123    5659 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889126    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889128    5659 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889131    5659 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889133    5659 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889136    5659 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889139    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889141    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889144    5659 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889146    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889149    5659 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889152    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889154    5659 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889157    5659 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889160    5659 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889162    5659 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889165    5659 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889167    5659 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:27.903877 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889170    5659 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889172    5659 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889175    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889177    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889180    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889184    5659 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889187    5659 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889190    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889192    5659 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889195    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889198    5659 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889201    5659 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889203    5659 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889206    5659 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889209    5659 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889212    5659 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889214    5659 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889217    5659 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889219    5659 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:27.904866 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889222    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889225    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889227    5659 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889230    5659 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889233    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889236    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889238    5659 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889241    5659 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889243    5659 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889246    5659 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889249    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889252    5659 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889254    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889256    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889259    5659 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889261    5659 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889264    5659 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889267    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889269    5659 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:27.905716 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889272    5659 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889275    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.889300    5659 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889447    5659 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889452    5659 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889455    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889459    5659 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889462    5659 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889465    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889467    5659 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889470    5659 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889473    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889475    5659 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889493    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889497    5659 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:27.906525 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889500    5659 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889502    5659 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889505    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889507    5659 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889510    5659 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889513    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889515    5659 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889518    5659 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889520    5659 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889524    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889527    5659 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889529    5659 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889532    5659 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889535    5659 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889537    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889540    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889542    5659 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889545    5659 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889547    5659 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889550    5659 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:27.907200 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889553    5659 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889569    5659 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889572    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889576    5659 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889580    5659 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889582    5659 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889585    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889587    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889590    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889592    5659 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889595    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889598    5659 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889600    5659 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889603    5659 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889606    5659 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889608    5659 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889611    5659 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889614    5659 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889616    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:27.908199 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889619    5659 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889622    5659 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889625    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889629    5659 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889632    5659 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889634    5659 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889637    5659 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889640    5659 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889642    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889645    5659 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889647    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889650    5659 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889652    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889655    5659 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889657    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889660    5659 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889662    5659 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889665    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889667    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889670    5659 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:27.909141 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889672    5659 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889675    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889677    5659 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889680    5659 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889683    5659 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889686    5659 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889688    5659 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889691    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889694    5659 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889696    5659 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889699    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889702    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889704    5659 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889707    5659 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889709    5659 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889711    5659 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889715    5659 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889717    5659 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889720    5659 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889722    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:27.910076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889725    5659 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889727    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889730    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889732    5659 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889736    5659 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889739    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889743    5659 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889745    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889748    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889751    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889754    5659 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889756    5659 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889759    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889762    5659 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889764    5659 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:27.911075 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.889787    5659 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889815    5659 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889819    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889823    5659 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889826    5659 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889828    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889831    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889834    5659 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889836    5659 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889839    5659 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889841    5659 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889845    5659 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889847    5659 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889850    5659 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889856    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889859    5659 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889861    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889864    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889866    5659 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889870    5659 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:27.911819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889874    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889876    5659 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889881    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889884    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889887    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889889    5659 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889892    5659 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889895    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889897    5659 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889900    5659 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889904    5659 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889907    5659 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889911    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889914    5659 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889917    5659 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889920    5659 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889923    5659 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889925    5659 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889928    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:27.912683 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889931    5659 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889933    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889936    5659 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889938    5659 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889941    5659 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889943    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889946    5659 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889948    5659 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889951    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889954    5659 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889957    5659 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889959    5659 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889962    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889965    5659 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889967    5659 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889971    5659 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889974    5659 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889976    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889979    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889982    5659 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:27.913446 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889984    5659 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889987    5659 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889990    5659 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889993    5659 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889995    5659 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.889998    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890001    5659 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890003    5659 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890006    5659 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890008    5659 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890011    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890013    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890016    5659 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890018    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890021    5659 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890023    5659 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890026    5659 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890029    5659 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890031    5659 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:27.914243 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890034    5659 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890036    5659 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890039    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890042    5659 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890045    5659 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890047    5659 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890050    5659 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890052    5659 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890056    5659 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890059    5659 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890062    5659 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890064    5659 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890068    5659 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890071    5659 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890073    5659 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890076    5659 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890078    5659 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890081    5659 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890084    5659 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890086    5659 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:27.914819 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890089    5659 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890091    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890094    5659 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890096    5659 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890099    5659 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890101    5659 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890104    5659 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890108    5659 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:27.890110    5659 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.890164    5659 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.891317    5659 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.891758    5659 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.892078    5659 server.go:1014] "Starting client certificate rotation"
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.892382    5659 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.892610    5659 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:31:16 +0000 UTC" deadline="2028-05-02 19:44:16.369016025 +0000 UTC"
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.892667    5659 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13104h6m48.476352059s"
Nov 03 19:37:27.915308 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.893018    5659 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:37:27.915791 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.894076    5659 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:37:27.915791 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.895343    5659 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:37:27.915791 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.897168    5659 log.go:25] "Validated CRI v1 image API"
Nov 03 19:37:27.915791 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.897971    5659 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:37:27.915791 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.901165    5659 fs.go:135] Filesystem UUIDs: map[2cb1254e-94a4-4d99-ae76-a2a48d1ddd11:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c503ccbe-cc5f-48b3-860d-24aa5b9a5e86:/dev/nvme0n1p4]
Nov 03 19:37:27.916045 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.901195    5659 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a3ad723292a1d970767de948de47caa2775987494c8849d2c01687519c75cdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a3ad723292a1d970767de948de47caa2775987494c8849d2c01687519c75cdb/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c2740c722e154dea4fa7c3ab8387754bcf4654e4dac8361ab8b0c0f5ea0effc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c2740c722e154dea4fa7c3ab8387754bcf4654e4dac8361ab8b0c0f5ea0effc/userdata/shm major:0 minor:374 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1eb57135de286a127233d228fa622464972983143f5729150279ebd5289647dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1eb57135de286a127233d228fa622464972983143f5729150279ebd5289647dc/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2407f8e452cd56a6af3ec945e8304de673e2a28c30806b7d4aaf0b6f5efa66b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2407f8e452cd56a6af3ec945e8304de673e2a28c30806b7d4aaf0b6f5efa66b6/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41dfb349eab98c7c8e5cfe4b15803d9fb5393310c0f09b38bcf56d1d9f43273b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41dfb349eab98c7c8e5cfe4b15803d9fb5393310c0f09b38bcf56d1d9f43273b/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48cbaded1c0107af1ae8d3c350d4a35a96fbef00ee0ca74b54e3b62b45fff466/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48cbaded1c0107af1ae8d3c350d4a35a96fbef00ee0ca74b54e3b62b45fff466/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/677dff29918d74053b715438e63808bad34068d201bb5f1b81f9bbdf43f706b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/677dff29918d74053b715438e63808bad34068d201bb5f1b81f9bbdf43f706b5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6bb512376fd1e2183d02c940760b7c11982fac4352a61012aa9f93946a6f0d8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6bb512376fd1e2183d02c940760b7c11982fac4352a61012aa9f93946a6f0d8f/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e0e44d033431d15a3b9e1eea482cc99809e57e9b9bfc4e8be326bb66ea0be06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e0e44d033431d15a3b9e1eea482cc99809e57e9b9bfc4e8be326bb66ea0be06/userdata/shm major:0 minor:269 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/799e49e8455b73a0798a741ae56920d0b8edac42e1b47a575aa8cbe1bf30263d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/799e49e8455b73a0798a741ae56920d0b8edac42e1b47a575aa8cbe1bf30263d/userdata/shm major:0 minor:407 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e9861ceb2125342bfad482387a49b5d772d504601db52983f25766fb5420c3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e9861ceb2125342bfad482387a49b5d772d504601db52983f25766fb5420c3a/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ef65d4b1d6ce321edb022e352a3ca81c3f22c484c110a9d001c1ae857a2db63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ef65d4b1d6ce321edb022e352a3ca81c3f22c484c110a9d001c1ae857a2db63/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/910af620224e6213f2d144032a30182959fcbcef6f0ba1a8847a8801c630a42e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/910af620224e6213f2d144032a30182959fcbcef6f0ba1a8847a8801c630a42e/userdata/shm major:0 minor:403 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9347aba7b20c207e2b7a373af25c5a17360da4d60d39de80b696be49f5d36bda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9347aba7b20c207e2b7a373af25c5a17360da4d60d39de80b696be49f5d36bda/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96328b245f60ae7829d009f60842f5c5c34b7a4dd849bad09594465546d487cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96328b245f60ae7829d009f60842f5c5c34b7a4dd849bad09594465546d487cf/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3c0675a6699e8ee3b795bacd4c367cb7109896eeb463faa73679150e9dc62e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3c0675a6699e8ee3b795bacd4c367cb7109896eeb463faa73679150e9dc62e6/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a85fa15687989263b0d138222aa85fea42b528f5b63b41cf9c82060381afeeb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a85fa15687989263b0d138222aa85fea42b528f5b63b41cf9c82060381afeeb8/userdata/shm major:0 minor:398 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9136d471b1ca4ba4499fedde7fdb262c5c0e8dfc6b492c43a2581ac3c51c1c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9136d471b1ca4ba4499fedde7fdb262c5c0e8dfc6b492c43a2581ac3c51c1c5/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af0d3cf6a78f293142be0e4c8b640931ec1f25ed2946d8db528a4200494f360d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af0d3cf6a78f293142be0e4c8b640931ec1f25ed2946d8db528a4200494f360d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b37a78448ca767d1b95c0194bee981aebc10b55e1732396d0a1b7bee4b2be2a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b37a78448ca767d1b95c0194bee981aebc10b55e1732396d0a1b7bee4b2be2a9/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6ca5648eb0ccce5b434f08a60c23e27b7fbd452db9df952ad10aa73317cb85a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6ca5648eb0ccce5b434f08a60c23e27b7fbd452db9df952ad10aa73317cb85a/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdcb2de077db9dfd6db8c0ca41da578a27118ffae4f6f3e03a924e235d93f950/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdcb2de077db9dfd6db8c0ca41da578a27118ffae4f6f3e03a924e235d93f950/userdata/shm major:0 minor:404 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d391ce127503ef4acd3720aeab598cc6b49a1b1c37bfbff9e968cb7e51f55432/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d391ce127503ef4acd3720aeab598cc6b49a1b1c37bfbff9e968cb7e51f55432/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4573b6c07d46d668269dd08759155a50b743f1ab4fae0d8f764e63052809f55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4573b6c07d46d668269dd08759155a50b743f1ab4fae0d8f764e63052809f55/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa4b784afcde48d129fed5739bd18e1912366eb56456f5af792846354f6c0389/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa4b784afcde48d129fed5739bd18e1912366eb56456f5af792846354f6c0389/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/124e9f93-f05c-4eee-8054-ac67754550c3/volumes/kubernetes.io~projected/kube-api-access-l4ndj:{mountpoint:/var/lib/kubelet/pods/124e9f93-f05c-4eee-8054-ac67754550c3/volumes/kubernetes.io~projected/kube-api-access-l4ndj major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/146c892e-d6ef-4062-94c4-6dae80875952/volumes/kubernetes.io~projected/kube-api-access-vp2b4:{mountpoint:/var/lib/kubelet/pods/146c892e-d6ef-4062-94c4-6dae80875952/volumes/kubernetes.io~projected/kube-api-access-vp2b4 major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/146c892e-d6ef-4062-94c4-6dae80875952/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/146c892e-d6ef-4062-94c4-6dae80875952/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/146c892e-d6ef-4062-94c4-6dae80875952/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/146c892e-d6ef-4062-94c4-6dae80875952/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ce3bacc-fa93-4880-983e-ac1b2ae7e488/volumes/kubernetes.io~projected/kube-api-access-xrmfq:{mountpoint:/var/lib/kubelet/pods/1ce3bacc-fa93-4880-983e-ac1b2ae7e488/volumes/kubernetes.io~projected/kube-api-access-xrmfq major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ce3bacc-fa93-4880-983e-ac1b2ae7e488/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/1ce3bacc-fa93-4880-983e-ac1b2ae7e488/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ce3bacc-fa93-4880-983e-ac1b2ae7e488/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/1ce3bacc-fa93-4880-983e-ac1b2ae7e488/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b6a3864-3172-40f0-9ab4-7327e2bf461c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2b6a3864-3172-40f0-9ab4-7327e2bf461c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b6a3864-3172-40f0-9ab4-7327e2bf461c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2b6a3864-3172-40f0-9ab4-7327e2bf461c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b6a3864-3172-40f0-9ab4-7327e2bf461c/volumes/kubernetes.io~projected/kube-api-access-v8ssv:{mountpoint:/var/lib/kubelet/pods/2b6a3864-3172-40f0-9ab4-7327e2bf461c/volumes/kubernetes.io~projected/kube-api-access-v8ssv major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d6f9087-55e4-4e30-905a-3a1493336ded/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2d6f9087-55e4-4e30-905a-3a1493336ded/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d6f9087-55e4-4e30-905a-3a1493336ded/volumes/kubernetes.io~projected/kube-api-access-thm8g:{mountpoint:/var/lib/kubelet/pods/2d6f9087-55e4-4e30-905a-3a1493336ded/volumes/kubernetes.io~projected/kube-api-access-thm8g major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d6f9087-55e4-4e30-905a-3a1493336ded/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2d6f9087-55e4-4e30-905a-3a1493336ded/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4af39c8b-d06b-4569-bc27-ca478cc4852c/volumes/kubernetes.io~projected/kube-api-access-wn6n4:{mountpoint:/var/lib/kubelet/pods/4af39c8b-d06b-4569-bc27-ca478cc4852c/volumes/kubernetes.io~projected/kube-api-access-wn6n4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d03e8db-60e2-4844-94c3-78f02bc19100/volumes/kubernetes.io~projected/kube-api-access-m8wnj:{mountpoint:/var/lib/kubelet/pods/4d03e8db-60e2-4844-94c3-78f02bc19100/volumes/kubernetes.io~projected/kube-api-access-m8wnj major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d03e8db-60e2-4844-94c3-78f02bc19100/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4d03e8db-60e2-4844-94c3-78f02bc19100/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d03e8db-60e2-4844-94c3-78f02bc19100/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/4d03e8db-60e2-4844-94c3-78f02bc19100/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dd8ccdd-1987-4200-8963-983dc4fd445d/volumes/kubernetes.io~projected/kube-api-access-nngxb:{mountpoint:/var/lib/kubelet/pods/4dd8ccdd-1987-4200-8963-983dc4fd445d/volumes/kubernetes.io~projected/kube-api-access-nngxb major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51cdd330-78ed-418e-8bdb-1f080d7c2b11/volumes/kubernetes.io~projected/kube-api-access-fs4rw:{mountpoint:/var/lib/kubelet/pods/51cdd330-78ed-418e-8bdb-1f080d7c2b11/volumes/kubernetes.io~projected/kube-api-access-fs4rw major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51cdd330-78ed-418e-8bdb-1f080d7c2b11/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/51cdd330-78ed-418e-8bdb-1f080d7c2b11/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51cdd330-78ed-418e-8bdb-1f080d7c2b11/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/51cdd330-78ed-418e-8bdb-1f080d7c2b11/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volume-subpaths/web-config/config-reloader/2 major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volume-subpaths/web-config/prometheus/15 major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~empty-dir/config-out major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~projected/kube-api-access-d6k67:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~projected/kube-api-access-d6k67 major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~projected/tls-assets major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/config major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/web-config major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/658b22ed-1324-4c81-b112-f051fdd3baa4/volumes/kubernetes.io~projected/kube-api-access-lbqjz:{mountpoint:/var/lib/kubelet/pods/658b22ed-1324-4c81-b112-f051fdd3baa4/volumes/kubernetes.io~projected/kube-api-access-lbqjz major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/658b22ed-1324-4c81-b112-f051fdd3baa4/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/658b22ed-1324-4c81-b112-f051fdd3baa4/volumes/kubernetes.io~secret/metrics-certs major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/674ad509-03fd-4f35-bc70-3a3602457de9/volumes/kubernetes.io~projected/kube-api-access-nbk8q:{mountpoint:/var/lib/kubelet/pods/674ad509-03fd-4f35-bc70-3a3602457de9/volumes/kubernetes.io~projected/kube-api-access-nbk8q major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/674ad509-03fd-4f35-bc70-3a3602457de9/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/674ad509-03fd-4f35-bc70-3a3602457de9/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74569270-9489-4c9f-b51e-d819081baa5d/volumes/kubernetes.io~projected/kube-api-access-ctw8s:{mountpoint:/var/lib/kubelet/pods/74569270-9489-4c9f-b51e-d819081baa5d/volumes/kubernetes.io~projected/kube-api-access-ctw8s major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74569270-9489-4c9f-b51e-d819081baa5d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/74569270-9489-4c9f-b51e-d819081baa5d/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~projected/kube-api-access-48pr9:{mountpoint:/var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~projected/kube-api-access-48pr9 major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99464aea-b429-4992-b4dd-ad48c7d99633/volumes/kubernetes.io~projected/kube-api-access-7knzb:{mountpoint:/var/lib/kubelet/pods/99464aea-b429-4992-b4dd-ad48c7d99633/volumes/kubernetes.io~projected/kube-api-access-7knzb major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99464aea-b429-4992-b4dd-ad48c7d99633/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/99464aea-b429-4992-b4dd-ad48c7d99633/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99464aea-b429-4992-b4dd-ad48c7d99633/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/99464aea-b429-4992-b4dd-ad48c7d99633/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0edbce3-cc93-410d-b82e-27e13cb6412c/volumes/kubernetes.io~projected/kube-api-access-plbb9:{mountpoint:/var/lib/kubelet/pods/a0edbce3-cc93-410d-b82e-27e13cb6412c/volumes/kubernetes.io~projected/kube-api-access-plbb9 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0edbce3-cc93-410d-b82e-27e13cb6412c/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/a0edbce3-cc93-410d-b82e-27e13cb6412c/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0edbce3-cc93-410d-b82e-27e13cb6412c/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/a0edbce3-cc93-410d-b82e-27e13cb6412c/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0edbce3-cc93-410d-b82e-27e13cb6412c/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/a0edbce3-cc93-410d-b82e-27e13cb6412c/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad627830-62cd-4f5b-bb63-76b5944b91ff/volumes/kubernetes.io~projected/kube-api-access-j68r2:{mountpoint:/var/lib/kubelet/pods/ad627830-62cd-4f5b-bb63-76b5944b91ff/volumes/kubernetes.io~projected/kube-api-access-j68r2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae317c95-3690-4603-a062-652384d87f55/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ae317c95-3690-4603-a062-652384d87f55/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c43e3b1f-9e9c-4768-8c45-e779762180bd/volumes/kubernetes.io~projected/kube-api-access-ttmj2:{mountpoint:/var/lib/kubelet/pods/c43e3b1f-9e9c-4768-8c45-e779762180bd/volumes/kubernetes.io~projected/kube-api-access-ttmj2 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8236a6c-f408-4996-906d-a73a8cb0069d/volumes/kubernetes.io~projected/kube-api-access-sj9m7:{mountpoint:/var/lib/kubelet/pods/c8236a6c-f408-4996-906d-a73a8cb0069d/volumes/kubernetes.io~projected/kube-api-access-sj9m7 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8236a6c-f408-4996-906d-a73a8cb0069d/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/c8236a6c-f408-4996-906d-a73a8cb0069d/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8236a6c-f408-4996-906d-a73a8cb0069d/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/c8236a6c-f408-4996-906d-a73a8cb0069d/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e50201a6-9ca6-4ff0-9e55-cc324745331e/volumes/kubernetes.io~projected/kube-api-access-dfpsh:{mountpoint:/var/lib/kubelet/pods/e50201a6-9ca6-4ff0-9e55-cc324745331e/volumes/kubernetes.io~projected/kube-api-access-dfpsh major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e50201a6-9ca6-4ff0-9e55-cc324745331e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/e50201a6-9ca6-4ff0-9e55-cc324745331e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e79d1e48-bb87-4a1b-9b88-95a1551e959c/volumes/kubernetes.io~projected/kube-api-access-28xl5:{mountpoint:/var/lib/kubelet/pods/e79d1e48-bb87-4a1b-9b88-95a1551e959c/volumes/kubernetes.io~projected/kube-api-access-28xl5 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc6cb168-a115-4918-a092-bc340accd868/volumes/kubernetes.io~projected/kube-api-access-zxm9f:{mountpoint:/var/lib/kubelet/pods/fc6cb168-a115-4918-a092-bc340accd868/volumes/kubernetes.io~projected/kube-api-access-zxm9f major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6a0525039df18ed2b1c4cb3404da6a15d57c7a25510e56927d8ba4c08ce566a6/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ed4684776901704d7c87d6f691b2d47c5c9c40fd1a1f80eeb401a635485aa704/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/4f825ba23b937658d85eec839f565e715eb00c0c33f64536fd978fee5f6b1978/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/0fd97c05cf23681962550bfac3182fd7e0968b422a235096806cd8e90fc4bd67/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1ff568134446025ec4ef698a90f90dc01d95619702dfac94bb943045113e003a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2f180a5547925bd23fc4e63316721fdeeeacccf7446985a004f41347d29aea5d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7b8114257ecadac3bc2cb270dda45f090651c57cf16942bef0c00cc49291af44/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b3b8efe3dd1efacd28c5caeb878b96877956dba441a1c171bed8391f8effcd74/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/53ec4c6f5e77457b5da4b97c0746d74aee7036d536619d136d2bbf8ceb6caab2/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b243931abf639f3c1d3d5c4322064cf32f753a3c9a02006f3fd1c59ea4ac0e54/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/4ebb0ffb0b7f600b15ed908e44c4a681557e488206bda917b1276c14f456df76/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/7b0279094d3d39d6335dde51ddf03b3c6a8262a504c6b6f64218d26acacd4186/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3f48447aac8d6bcad718a2456cd9c352468ad8b7253436b467cd907d8352c87f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/95ee60ab71d9385b8d7b7d9c866b98f8735f0db4c4bd72d8809222508169aec9/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/22adf38af33d97f7386a8a5766892033cd2caeb5e86e3e9c15ada09cdf5a772e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/fcfc50af1da0d2f92ffeeccadc883dfc081e2a878269bd6e105f786c901e72df/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/01e90d054fdde4ce630af18fd5d766b50f98d1cf7433a73685b84f5f3402bdb0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/59757d226bee70ddee66d8dd8ef7bbc78aa18d34cd817ef89ccda3e53c30001f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5e0d5d3ee53f4cdab4df31b7be9eb4b7c7f63f5661bc63033370c233f9c8b68a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cf3f1e6a4f146162d6c36be352f790043d4158d33e7fd2c2a299f1514dfbb5a7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/938a4b280f449a6d2d27db1d76c203ca9ac7123a05e37cba0d954c3c06eb2b66/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/d74955bea0b493609bf2f49bf10a203b0d895bc23b0eb8443b670a2753abc6c7/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/58ef25e09ed5ae861342f6be4f6b42db37ff7f29fbfdbae3948d639b956003b7/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/b51e18c3e5470b0a0648597e40e03d0c3da252a452b51274a6c7ef43acf91140/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/767c13f1b95d6c7422caa9d88557ec36b73aa526a904833a7a42c837a2dfa9dc/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/7309474cdd7d2caa0c262bf4de658605892b3000711ce66213066a571375196d/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/e2312d16af37631666b373a54133b0bb24d000c13ffa8bde917143a4fbe71b1a/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/76b91b3c446ed17d30865194e2070801d8d6d1eb634906da41ecaee9ead1e48f/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/3a6a2ac58ce3c54c21caa473ce8c2fbc1060ba3969d4259c2065b88b1fe4d173/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/55c9a554d8feef0d08bb98fac71c541be5811c8ceb74f2da52b675f94ebf0c72/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/28d8031a2f0ebeaf2fe019a9a6627817b8d7c7c86c2ec609cc59e5ba6e02ae03/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/0df0138478559387e3b120772d2fe7834c3c19a5312ea5e0da30dbb7292d83ec/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/6cdbc736d09e2df552c90e5b235d91755ab33fb145befb46a7afa8fe199c8486/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/218e5a0beed3af566f4d88e0925737dbdae75372bac04985feedc606e71f3157/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/91f85bb7fc8f1a3ed0ac62b1c81fdf1515076a648aa318a387df768851f99fc3/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/47016116d52ebeafb4787638b805f9b5864c64a4c317452e070855b48810cfbe/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/d42b83c5752424c6b80db03eefcbf71c50995149676377654780a7aceb94a4ba/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/de918822e6eeee91df36417839db4a9132c2bf837bee3a6143439eb32a8bb8bd/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/bb8c8b2cde4d953e5fbbb0181b40530f84d754bba3eafba6280e088eca34e788/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/f6c118faefa58a44dda25777aba472be11a009a34f1a37f9049cf5fefeef48f0/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/254c5eac7095d2cd14f165c2fad6d2f431ab2a980153653f9f7c9375b5cd5f10/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/a62ec59ce27b46e78335b84a13359ea83d46253b7e1d88c5f4bba7062f15f33b/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/54bdb38ac9ab8116d3ad0385f262fbe66d58658748c88431f7c05d103743cf1b/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/b3bb142f46c7885959e2fe092eb1ac8435f3dbe315db420ac3407033d788d6aa/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/3c2476ddd9bee7e4c80e200a17d4cb84d3f4d83857968e3d7172e6012c64f199/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/9cc39ac436d9dac25f6a000424201960b5cba7bc0b14e5ec56873eb21834e6c2/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/4cf3b048080031464442889b79d2173b238f1f1cb32f50761c2aee8cd67740c1/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/eb961571833ebad82aadc8eb8763dbd7938a6e84eff4d5c949237210db58de8e/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/8ce4191f4a9f590d2dc19c256c525b5f08e349e0cb275bec645463001f3093c7/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/e46bb54e5f4627eb9b9ece271024b2469698c0f2c8da8c62521eff2f9818d2b4/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/c00bf4a2952bde6cc9143b7054fe50270f4c550c3526615daa87f3bee330ade7/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/078cbf7f694a7a9ee823a862a4e8be9c587d066928773d2f150a26bf561d7915/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/7ec2a013d7601e61947617c9c0720f78a9f3a9a28e8cdce8bda7ca325966a6c9/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/e47306290207efc5d62d8e3309c96281b709c1a624223f57f9c24a2b9f6f4161/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/681d02cbdf8c8548e24b6546cb2d2ac290d02af9f6e6e41544950dd4594f3446/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/77a36e172e2425b952cc7d36d0d3410b70cb8abe1176bab4b4aedddcc9a25773/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/9d1160824170bf12ffaada6c1bcf8219924a3fa95cf3ba3665fcbafc41b91cf9/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/8dc491072f9a2d32e232e306819536a368118786a7f137b46cc9f65c4176919a/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/80594693641c19b4e92cbf345d6c7c0a888f254ba3cf16d35aba8056a4c98b63/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/f98fc90d99ebd3153260ea7401349c06aeceae07c66a3638930e02ab0960f14f/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/d4634f1c590acab4c07464512693ecf98a2bb6aa28d2f207a6e7af89ec948309/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/fd363f2194810f49cf3c296e7715087f3c5411440f8cbb2c1a1b6a9295219a0f/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/5bb39802c07b6796c923e52308b7b57c4754314c232922f5b72720a56256303e/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/ddf88c132f806f15b3d27aec1edd9c3cc2c55a9f68edec384c70e6ef73898bea/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/c164235353d0d786824c7f8a945cd4a3a1dda09cb6c4e95ab0357acd1ce53852/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/9bb50e477c2bad88ac2570642aa52e38c13d3d6eacc3a9c961ff12d93256e897/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ea27ad58f086e0e9b9bc67b956b3b9239d9b71f455a25022237c6bd59bbcd3e3/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/38f8aac61d8ce15bc5f9763f9082c39e8d33d770636d5909fa65bca746beeeb6/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/dededed5d96b59203b3f48d02fbd0a91e7c4198e5766f1a83746624eed3af340/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/61402e5115e290f0997378f058da549190c596663705df892ab05b0d59a5fa6e/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/f262a7d6616b3b3a9b8706bb770e122913253465251ae23a18dc7eae37c36f0d/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/a7353832093ff29c7315ab4d6df91a99a03f95fe5ed60d76cd4a2d980656bf90/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/9b82d28e5c25e53659c980b537779d9cc35d0f5fe41a8626508601a407301568/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/643f765546cd4da6cd330e7981d4ae8000d826ef96436118138c7078ac456a35/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/dfc3ea49c252c2739574bee238ff0175ee7904adba804d2d9a85c2eb60a70a58/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.913173    5659 manager.go:217] Machine: {Timestamp:2026-11-03 19:37:27.911982046 +0000 UTC m=+0.066884457 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099353 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f62831c0fb56fe29035638d88aaea SystemUUID:ec2f6283-1c0f-b56f-e290-35638d88aaea BootID:27ecc72c-c281-463e-a55c-4a8b2a0976c2 Filesystems:[{Device:/var/lib/kubelet/pods/e50201a6-9ca6-4ff0-9e55-cc324745331e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8236a6c-f408-4996-906d-a73a8cb0069d/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:285 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0edbce3-cc93-410d-b82e-27e13cb6412c/volumes/kubernetes.io~projected/kube-api-access-plbb9 DeviceMajor:0 DeviceMinor:319 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99464aea-b429-4992-b4dd-ad48c7d99633/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:372 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/677dff29918d74053b715438e63808bad34068d201bb5f1b81f9bbdf43f706b5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3c0675a6699e8ee3b795bacd4c367cb7109896eeb463faa73679150e9dc62e6/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d391ce127503ef4acd3720aeab598cc6b49a1b1c37bfbff9e968cb7e51f55432/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99464aea-b429-4992-b4dd-ad48c7d99633/volumes/kubernetes.io~projected/kube-api-access-7knzb DeviceMajor:0 DeviceMinor:373 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6ca5648eb0ccce5b434f08a60c23e27b7fbd452db9df952ad10aa73317cb85a/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af0d3cf6a78f293142be0e4c8b640931ec1f25ed2946d8db528a4200494f360d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:338 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4af39c8b-d06b-4569-bc27-ca478cc4852c/volumes/kubernetes.io~projected/kube-api-access-wn6n4 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9347aba7b20c207e2b7a373af25c5a17360da4d60d39de80b696be49f5d36bda/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/124e9f93-f05c-4eee-8054-ac67754550c3/volumes/kubernetes.io~projected/kube-api-access-l4ndj DeviceMajor:0 DeviceMinor:402 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0edbce3-cc93-410d-b82e-27e13cb6412c/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:317 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0edbce3-cc93-410d-b82e-27e13cb6412c/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:318 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:346 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a3ad723292a1d970767de948de47caa2775987494c8849d2c01687519c75cdb/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e9861ceb2125342bfad482387a49b5d772d504601db52983f25766fb5420c3a/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d03e8db-60e2-4844-94c3-78f02bc19100/volumes/kubernetes.io~projected/kube-api-access-m8wnj DeviceMajor:0 DeviceMinor:265 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48cbaded1c0107af1ae8d3c350d4a35a96fbef00ee0ca74b54e3b62b45fff466/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51cdd330-78ed-418e-8bdb-1f080d7c2b11/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:344 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~projected/kube-api-access-d6k67 DeviceMajor:0 DeviceMinor:347 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1ce3bacc-fa93-4880-983e-ac1b2ae7e488/volumes/kubernetes.io~projected/kube-api-access-xrmfq DeviceMajor:0 DeviceMinor:397 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d03e8db-60e2-4844-94c3-78f02bc19100/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/146c892e-d6ef-4062-94c4-6dae80875952/volumes/kubernetes.io~projected/kube-api-access-vp2b4 DeviceMajor:0 DeviceMinor:268 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e50201a6-9ca6-4ff0-9e55-cc324745331e/volumes/kubernetes.io~projected/kube-api-access-dfpsh DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2407f8e452cd56a6af3ec945e8304de673e2a28c30806b7d4aaf0b6f5efa66b6/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8236a6c-f408-4996-906d-a73a8cb0069d/volumes/kubernetes.io~projected/kube-api-access-sj9m7 DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:325 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa4b784afcde48d129fed5739bd18e1912366eb56456f5af792846354f6c0389/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6bb512376fd1e2183d02c940760b7c11982fac4352a61012aa9f93946a6f0d8f/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ce3bacc-fa93-4880-983e-ac1b2ae7e488/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:396 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/799e49e8455b73a0798a741ae56920d0b8edac42e1b47a575aa8cbe1bf30263d/userdata/shm DeviceMajor:0 DeviceMinor:407 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:337 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74569270-9489-4c9f-b51e-d819081baa5d/volumes/kubernetes.io~projected/kube-api-access-ctw8s DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:337 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/658b22ed-1324-4c81-b112-f051fdd3baa4/volumes/kubernetes.io~projected/kube-api-access-lbqjz DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51cdd330-78ed-418e-8bdb-1f080d7c2b11/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1eb57135de286a127233d228fa622464972983143f5729150279ebd5289647dc/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:342 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae317c95-3690-4603-a062-652384d87f55/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:400 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0edbce3-cc93-410d-b82e-27e13cb6412c/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:312 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/910af620224e6213f2d144032a30182959fcbcef6f0ba1a8847a8801c630a42e/userdata/shm DeviceMajor:0 DeviceMinor:403 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/674ad509-03fd-4f35-bc70-3a3602457de9/volumes/kubernetes.io~projected/kube-api-access-nbk8q DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/146c892e-d6ef-4062-94c4-6dae80875952/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:267 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:324 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/658b22ed-1324-4c81-b112-f051fdd3baa4/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:401 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b6a3864-3172-40f0-9ab4-7327e2bf461c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2d6f9087-55e4-4e30-905a-3a1493336ded/volumes/kubernetes.io~projected/kube-api-access-thm8g DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:326 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~projected/kube-api-access-48pr9 DeviceMajor:0 DeviceMinor:328 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:337 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1ce3bacc-fa93-4880-983e-ac1b2ae7e488/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:392 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdcb2de077db9dfd6db8c0ca41da578a27118ffae4f6f3e03a924e235d93f950/userdata/shm DeviceMajor:0 DeviceMinor:404 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96328b245f60ae7829d009f60842f5c5c34b7a4dd849bad09594465546d487cf/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b37a78448ca767d1b95c0194bee981aebc10b55e1732396d0a1b7bee4b2be2a9/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9136d471b1ca4ba4499fedde7fdb262c5c0e8dfc6b492c43a2581ac3c51c1c5/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d6f9087-55e4-4e30-905a-3a1493336ded/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc6cb168-a115-4918-a092-bc340accd868/volumes/kubernetes.io~projected/kube-api-access-zxm9f DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8236a6c-f408-4996-906d-a73a8cb0069d/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99464aea-b429-4992-b4dd-ad48c7d99633/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:368 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74569270-9489-4c9f-b51e-d819081baa5d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a85fa15687989263b0d138222aa85fea42b528f5b63b41cf9c82060381afeeb8/userdata/shm DeviceMajor:0 DeviceMinor:398 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93b47942-ccbf-43cd-8722-c21038f906f3/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:327 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/2b6a3864-3172-40f0-9ab4-7327e2bf461c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d6f9087-55e4-4e30-905a-3a1493336ded/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/146c892e-d6ef-4062-94c4-6dae80875952/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:266 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c2740c722e154dea4fa7c3ab8387754bcf4654e4dac8361ab8b0c0f5ea0effc/userdata/shm DeviceMajor:0 DeviceMinor:374 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b6a3864-3172-40f0-9ab4-7327e2bf461c/volumes/kubernetes.io~projected/kube-api-access-v8ssv DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4dd8ccdd-1987-4200-8963-983dc4fd445d/volumes/kubernetes.io~projected/kube-api-access-nngxb DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ef65d4b1d6ce321edb022e352a3ca81c3f22c484c110a9d001c1ae857a2db63/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e0e44d033431d15a3b9e1eea482cc99809e57e9b9bfc4e8be326bb66ea0be06/userdata/shm DeviceMajor:0 DeviceMinor:269 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:337 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:339 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e79d1e48-bb87-4a1b-9b88-95a1551e959c/volumes/kubernetes.io~projected/kube-api-access-28xl5 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51cdd330-78ed-418e-8bdb-1f080d7c2b11/volumes/kubernetes.io~projected/kube-api-access-fs4rw DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4573b6c07d46d668269dd08759155a50b743f1ab4fae0d8f764e63052809f55/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d03e8db-60e2-4844-94c3-78f02bc19100/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:264 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:343 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41dfb349eab98c7c8e5cfe4b15803d9fb5393310c0f09b38bcf56d1d9f43273b/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad627830-62cd-4f5b-bb63-76b5944b91ff/volumes/kubernetes.io~projected/kube-api-access-j68r2 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/674ad509-03fd-4f35-bc70-3a3602457de9/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c43e3b1f-9e9c-4768-8c45-e779762180bd/volumes/kubernetes.io~projected/kube-api-access-ttmj2 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0c2740c722e154d MacAddress:6e:df:8d:20:6f:09 Speed:10000 Mtu:8901} {Name:1eb57135de286a1 MacAddress:ea:47:67:f2:c6:90 Speed:10000 Mtu:8901} {Name:41dfb349eab98c7 MacAddress:0e:b4:83:7b:cf:88 Speed:10000 Mtu:8901} {Name:6e0e44d033431d1 MacAddress:3a:eb:79:86:bf:f7 Speed:10000 Mtu:8901} {Name:799e49e8455b73a MacAddress:26:a3:86:d4:70:f1 Speed:10000 Mtu:8901} {Name:8e9861ceb212534 MacAddress:96:15:5d:6b:2a:a8 Speed:10000 Mtu:8901} {Name:910af620224e621 MacAddress:f6:bb:6a:e7:ae:27 Speed:10000 Mtu:8901} {Name:96328b245f60ae7 MacAddress:46:d0:36:fd:40:5c Speed:10000 Mtu:8901} {Name:a3c0675a6699e8e MacAddress:02:79:79:9c:e2:59 Speed:10000 Mtu:8901} {Name:a85fa1568798926 MacAddress:fe:fc:ce:20:31:f4 Speed:10000 Mtu:8901} {Name:a9136d471b1ca4b MacAddress:4e:78:26:8f:06:ee Speed:10000 Mtu:8901} {Name:b37a78448ca767d MacAddress:06:76:54:8b:e9:fe Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cd:ea:19:8d Speed:0 Mtu:9001} {Name:br-int MacAddress:ae:d2:97:81:cc:a6 Speed:0 Mtu:8901} {Name:cdcb2de077db9df MacAddress:56:d4:01:94:6f:0f Speed:10000 Mtu:8901} {Name:d391ce127503ef4 MacAddress:36:6b:fb:72:22:eb Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cd:ea:19:8d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:56:3b:03:50:f4:06 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:de:f4:ad:03:37:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.914473    5659 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.914558    5659 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.914879    5659 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.914905    5659 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-249.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.915037    5659 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.915045    5659 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.915059    5659 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.915078    5659 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.915182    5659 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.915361    5659 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.915468    5659 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.915522    5659 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.915533    5659 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.915542    5659 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.915551    5659 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.915559    5659 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.916208    5659 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:37:27.916455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.916217    5659 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:37:27.917217 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.916771    5659 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:37:27.919371 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919355    5659 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:37:27.919371 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919369    5659 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:37:27.919492 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919472    5659 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 19:37:27.919696 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919684    5659 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:37:27.919736 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919714    5659 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:37:27.919818 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919807    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:37:27.919846 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919827    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:37:27.919846 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919841    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:37:27.919900 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919850    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:37:27.919900 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919856    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:37:27.919900 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919862    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:37:27.919900 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919868    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:37:27.919900 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919874    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:37:27.919900 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919881    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:37:27.919900 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919887    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:37:27.919900 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919899    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:37:27.920115 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919912    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:37:27.920115 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919939    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:37:27.920115 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.919948    5659 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:37:27.923798 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.923781    5659 server.go:1263] "Started kubelet"
Nov 03 19:37:27.923883 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.923841    5659 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:37:27.924297 ip-10-0-133-249 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:37:27.924454 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.924370    5659 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:37:27.925065 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.924852    5659 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:37:27.925212 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.925176    5659 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:37:27.928262 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.926140    5659 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:37:27.928697 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.928675    5659 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:37:27.929806 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.928825    5659 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:37:27.929806 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:27.928836    5659 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-249.us-east-2.compute.internal\" not found"
Nov 03 19:37:27.929806 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.928881    5659 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:37:27.929806 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.928888    5659 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:37:27.929806 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.929097    5659 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.930012    5659 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:31:18 +0000 UTC" deadline="2028-08-10 17:45:16.400462446 +0000 UTC"
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.930034    5659 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15502h7m48.470431732s"
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.930378    5659 factory.go:153] Registering CRI-O factory
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.930399    5659 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.930456    5659 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.930466    5659 factory.go:55] Registering systemd factory
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.930473    5659 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.930515    5659 factory.go:103] Registering Raw factory
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.930531    5659 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.930902    5659 manager.go:319] Starting recovery of all containers
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932270    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146c892e-d6ef-4062-94c4-6dae80875952" volumeName="kubernetes.io/configmap/146c892e-d6ef-4062-94c4-6dae80875952-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932324    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16b10ae2-69df-49f1-b0b3-77c98ee4cc3d" volumeName="kubernetes.io/configmap/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d-konnectivity-ca" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932339    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="674ad509-03fd-4f35-bc70-3a3602457de9" volumeName="kubernetes.io/projected/674ad509-03fd-4f35-bc70-3a3602457de9-kube-api-access-nbk8q" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932352    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0edbce3-cc93-410d-b82e-27e13cb6412c" volumeName="kubernetes.io/empty-dir/a0edbce3-cc93-410d-b82e-27e13cb6412c-audit-log" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932365    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146c892e-d6ef-4062-94c4-6dae80875952" volumeName="kubernetes.io/secret/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932378    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" volumeName="kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932392    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d6f9087-55e4-4e30-905a-3a1493336ded" volumeName="kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-ovnkube-config" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932406    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51cdd330-78ed-418e-8bdb-1f080d7c2b11" volumeName="kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-oauth-config" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932432    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b6a3864-3172-40f0-9ab4-7327e2bf461c" volumeName="kubernetes.io/empty-dir/2b6a3864-3172-40f0-9ab4-7327e2bf461c-tmp" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932445    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51cdd330-78ed-418e-8bdb-1f080d7c2b11" volumeName="kubernetes.io/projected/51cdd330-78ed-418e-8bdb-1f080d7c2b11-kube-api-access-fs4rw" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932458    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b47942-ccbf-43cd-8722-c21038f906f3" volumeName="kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-federate-client-tls" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932471    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad627830-62cd-4f5b-bb63-76b5944b91ff" volumeName="kubernetes.io/configmap/ad627830-62cd-4f5b-bb63-76b5944b91ff-serviceca" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932500    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" volumeName="kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-service-ca" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932520    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="658b22ed-1324-4c81-b112-f051fdd3baa4" volumeName="kubernetes.io/projected/658b22ed-1324-4c81-b112-f051fdd3baa4-kube-api-access-lbqjz" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932538    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="674ad509-03fd-4f35-bc70-3a3602457de9" volumeName="kubernetes.io/configmap/674ad509-03fd-4f35-bc70-3a3602457de9-config-volume" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932551    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99464aea-b429-4992-b4dd-ad48c7d99633" volumeName="kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-serving-cert" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932564    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc6cb168-a115-4918-a092-bc340accd868" volumeName="kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932578    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c43e3b1f-9e9c-4768-8c45-e779762180bd" volumeName="kubernetes.io/configmap/c43e3b1f-9e9c-4768-8c45-e779762180bd-iptables-alerter-script" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932590    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd8ccdd-1987-4200-8963-983dc4fd445d" volumeName="kubernetes.io/configmap/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-daemon-config" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932604    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-tls-assets" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932617    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932629    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e79d1e48-bb87-4a1b-9b88-95a1551e959c" volumeName="kubernetes.io/empty-dir/e79d1e48-bb87-4a1b-9b88-95a1551e959c-tmp-dir" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932641    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" volumeName="kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-serving-cert" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932653    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51cdd330-78ed-418e-8bdb-1f080d7c2b11" volumeName="kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-service-ca" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932673    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0edbce3-cc93-410d-b82e-27e13cb6412c" volumeName="kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-client-ca-bundle" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932686    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932699    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-grpc-tls" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932711    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8236a6c-f408-4996-906d-a73a8cb0069d" volumeName="kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-oauth-serving-cert" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932742    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" volumeName="kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-oauth-config" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932756    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b6a3864-3172-40f0-9ab4-7327e2bf461c" volumeName="kubernetes.io/projected/2b6a3864-3172-40f0-9ab4-7327e2bf461c-kube-api-access-v8ssv" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932770    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="674ad509-03fd-4f35-bc70-3a3602457de9" volumeName="kubernetes.io/empty-dir/674ad509-03fd-4f35-bc70-3a3602457de9-tmp-dir" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932784    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146c892e-d6ef-4062-94c4-6dae80875952" volumeName="kubernetes.io/empty-dir/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-textfile" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932797    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51cdd330-78ed-418e-8bdb-1f080d7c2b11" volumeName="kubernetes.io/empty-dir/51cdd330-78ed-418e-8bdb-1f080d7c2b11-tmp" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932810    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99464aea-b429-4992-b4dd-ad48c7d99633" volumeName="kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-oauth-config" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932823    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932835    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="124e9f93-f05c-4eee-8054-ac67754550c3" volumeName="kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932849    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d6f9087-55e4-4e30-905a-3a1493336ded" volumeName="kubernetes.io/projected/2d6f9087-55e4-4e30-905a-3a1493336ded-kube-api-access-thm8g" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932863    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51cdd330-78ed-418e-8bdb-1f080d7c2b11" volumeName="kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-serving-cert" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932876    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b6a3864-3172-40f0-9ab4-7327e2bf461c" volumeName="kubernetes.io/empty-dir/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-tuned" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932888    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-db" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932900    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b47942-ccbf-43cd-8722-c21038f906f3" volumeName="kubernetes.io/projected/93b47942-ccbf-43cd-8722-c21038f906f3-kube-api-access-48pr9" seLinuxMountContext=""
Nov 03 19:37:27.934807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932913    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0edbce3-cc93-410d-b82e-27e13cb6412c" volumeName="kubernetes.io/configmap/a0edbce3-cc93-410d-b82e-27e13cb6412c-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 19:37:27.937237 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937191    5659 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.932929    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8236a6c-f408-4996-906d-a73a8cb0069d" volumeName="kubernetes.io/empty-dir/c8236a6c-f408-4996-906d-a73a8cb0069d-tmp" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937350    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146c892e-d6ef-4062-94c4-6dae80875952" volumeName="kubernetes.io/configmap/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937372    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937388    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937401    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" volumeName="kubernetes.io/empty-dir/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-tmp" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937413    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d6f9087-55e4-4e30-905a-3a1493336ded" volumeName="kubernetes.io/secret/2d6f9087-55e4-4e30-905a-3a1493336ded-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937425    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="658b22ed-1324-4c81-b112-f051fdd3baa4" volumeName="kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937439    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b47942-ccbf-43cd-8722-c21038f906f3" volumeName="kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937451    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99464aea-b429-4992-b4dd-ad48c7d99633" volumeName="kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-oauth-serving-cert" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937463    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e50201a6-9ca6-4ff0-9e55-cc324745331e" volumeName="kubernetes.io/configmap/e50201a6-9ca6-4ff0-9e55-cc324745331e-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937476    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d03e8db-60e2-4844-94c3-78f02bc19100" volumeName="kubernetes.io/projected/4d03e8db-60e2-4844-94c3-78f02bc19100-kube-api-access-m8wnj" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937507    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937519    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99464aea-b429-4992-b4dd-ad48c7d99633" volumeName="kubernetes.io/projected/99464aea-b429-4992-b4dd-ad48c7d99633-kube-api-access-7knzb" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937532    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc6cb168-a115-4918-a092-bc340accd868" volumeName="kubernetes.io/projected/fc6cb168-a115-4918-a092-bc340accd868-kube-api-access-zxm9f" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937556    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d6f9087-55e4-4e30-905a-3a1493336ded" volumeName="kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-env-overrides" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937567    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d03e8db-60e2-4844-94c3-78f02bc19100" volumeName="kubernetes.io/configmap/4d03e8db-60e2-4844-94c3-78f02bc19100-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937579    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c43e3b1f-9e9c-4768-8c45-e779762180bd" volumeName="kubernetes.io/projected/c43e3b1f-9e9c-4768-8c45-e779762180bd-kube-api-access-ttmj2" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937590    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc6cb168-a115-4918-a092-bc340accd868" volumeName="kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937602    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937614    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-kube-api-access-d6k67" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937626    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99464aea-b429-4992-b4dd-ad48c7d99633" volumeName="kubernetes.io/empty-dir/99464aea-b429-4992-b4dd-ad48c7d99633-tmp" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937638    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937649    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b47942-ccbf-43cd-8722-c21038f906f3" volumeName="kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-secret-telemeter-client" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937660    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937671    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc6cb168-a115-4918-a092-bc340accd868" volumeName="kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937708    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d03e8db-60e2-4844-94c3-78f02bc19100" volumeName="kubernetes.io/secret/4d03e8db-60e2-4844-94c3-78f02bc19100-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937722    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99464aea-b429-4992-b4dd-ad48c7d99633" volumeName="kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937735    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99464aea-b429-4992-b4dd-ad48c7d99633" volumeName="kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-console-config" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937747    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e79d1e48-bb87-4a1b-9b88-95a1551e959c" volumeName="kubernetes.io/projected/e79d1e48-bb87-4a1b-9b88-95a1551e959c-kube-api-access-28xl5" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937758    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b47942-ccbf-43cd-8722-c21038f906f3" volumeName="kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937770    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae317c95-3690-4603-a062-652384d87f55" volumeName="kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937782    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8236a6c-f408-4996-906d-a73a8cb0069d" volumeName="kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-oauth-config" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937794    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146c892e-d6ef-4062-94c4-6dae80875952" volumeName="kubernetes.io/secret/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-tls" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937805    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d6f9087-55e4-4e30-905a-3a1493336ded" volumeName="kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937817    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4af39c8b-d06b-4569-bc27-ca478cc4852c" volumeName="kubernetes.io/projected/4af39c8b-d06b-4569-bc27-ca478cc4852c-kube-api-access-wn6n4" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937829    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="674ad509-03fd-4f35-bc70-3a3602457de9" volumeName="kubernetes.io/secret/674ad509-03fd-4f35-bc70-3a3602457de9-metrics-tls" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937844    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b47942-ccbf-43cd-8722-c21038f906f3" volumeName="kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937857    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16b10ae2-69df-49f1-b0b3-77c98ee4cc3d" volumeName="kubernetes.io/secret/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d-agent-certs" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937870    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937883    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b47942-ccbf-43cd-8722-c21038f906f3" volumeName="kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937895    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b47942-ccbf-43cd-8722-c21038f906f3" volumeName="kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-telemeter-client-tls" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937906    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0edbce3-cc93-410d-b82e-27e13cb6412c" volumeName="kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937918    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e50201a6-9ca6-4ff0-9e55-cc324745331e" volumeName="kubernetes.io/secret/e50201a6-9ca6-4ff0-9e55-cc324745331e-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937931    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937943    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0edbce3-cc93-410d-b82e-27e13cb6412c" volumeName="kubernetes.io/projected/a0edbce3-cc93-410d-b82e-27e13cb6412c-kube-api-access-plbb9" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937955    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8236a6c-f408-4996-906d-a73a8cb0069d" volumeName="kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-serving-cert" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937966    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74569270-9489-4c9f-b51e-d819081baa5d" volumeName="kubernetes.io/secret/74569270-9489-4c9f-b51e-d819081baa5d-cert" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937978    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0edbce3-cc93-410d-b82e-27e13cb6412c" volumeName="kubernetes.io/configmap/a0edbce3-cc93-410d-b82e-27e13cb6412c-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.937989    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d03e8db-60e2-4844-94c3-78f02bc19100" volumeName="kubernetes.io/secret/4d03e8db-60e2-4844-94c3-78f02bc19100-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938002    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51cdd330-78ed-418e-8bdb-1f080d7c2b11" volumeName="kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-config" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938013    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938025    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938038    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e50201a6-9ca6-4ff0-9e55-cc324745331e" volumeName="kubernetes.io/empty-dir/e50201a6-9ca6-4ff0-9e55-cc324745331e-data-volume" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938049    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8236a6c-f408-4996-906d-a73a8cb0069d" volumeName="kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-console-config" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938061    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" volumeName="kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-config" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938075    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" volumeName="kubernetes.io/projected/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-kube-api-access-xrmfq" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938088    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config-out" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938100    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-web-config" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938111    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99464aea-b429-4992-b4dd-ad48c7d99633" volumeName="kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-service-ca" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938123    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0edbce3-cc93-410d-b82e-27e13cb6412c" volumeName="kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938136    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8236a6c-f408-4996-906d-a73a8cb0069d" volumeName="kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938147    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad627830-62cd-4f5b-bb63-76b5944b91ff" volumeName="kubernetes.io/projected/ad627830-62cd-4f5b-bb63-76b5944b91ff-kube-api-access-j68r2" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938158    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8236a6c-f408-4996-906d-a73a8cb0069d" volumeName="kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-service-ca" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938169    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8236a6c-f408-4996-906d-a73a8cb0069d" volumeName="kubernetes.io/projected/c8236a6c-f408-4996-906d-a73a8cb0069d-kube-api-access-sj9m7" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938180    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51cdd330-78ed-418e-8bdb-1f080d7c2b11" volumeName="kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-oauth-serving-cert" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938193    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" volumeName="kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-metrics-client-certs" seLinuxMountContext=""
Nov 03 19:37:27.938233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938207    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" volumeName="kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-oauth-serving-cert" seLinuxMountContext=""
Nov 03 19:37:27.941441 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938858    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e50201a6-9ca6-4ff0-9e55-cc324745331e" volumeName="kubernetes.io/projected/e50201a6-9ca6-4ff0-9e55-cc324745331e-kube-api-access-dfpsh" seLinuxMountContext=""
Nov 03 19:37:27.941441 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938893    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="146c892e-d6ef-4062-94c4-6dae80875952" volumeName="kubernetes.io/projected/146c892e-d6ef-4062-94c4-6dae80875952-kube-api-access-vp2b4" seLinuxMountContext=""
Nov 03 19:37:27.941441 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938906    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd8ccdd-1987-4200-8963-983dc4fd445d" volumeName="kubernetes.io/configmap/4dd8ccdd-1987-4200-8963-983dc4fd445d-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:37:27.941441 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938961    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd8ccdd-1987-4200-8963-983dc4fd445d" volumeName="kubernetes.io/projected/4dd8ccdd-1987-4200-8963-983dc4fd445d-kube-api-access-nngxb" seLinuxMountContext=""
Nov 03 19:37:27.941441 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938977    5659 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74569270-9489-4c9f-b51e-d819081baa5d" volumeName="kubernetes.io/projected/74569270-9489-4c9f-b51e-d819081baa5d-kube-api-access-ctw8s" seLinuxMountContext=""
Nov 03 19:37:27.941441 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938990    5659 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:37:27.941441 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.938999    5659 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:37:27.941441 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.939282    5659 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:37:27.941441 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.940521    5659 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:37:27.947892 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.947871    5659 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 19:37:27.958897 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.958846    5659 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:37:27.960618 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.960594    5659 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:37:27.960618 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.960618    5659 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:37:27.960790 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.960638    5659 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:37:27.960790 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:27.960678    5659 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 19:37:27.966685 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.966659    5659 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:37:27.976777 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.976741    5659 generic.go:356] "Generic (PLEG): container finished" podID="146c892e-d6ef-4062-94c4-6dae80875952" containerID="76755129c608a9f4e6ee81f4c869ab3d98e6254f762f4a6c06b4ea2f85447378" exitCode=0
Nov 03 19:37:27.986768 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.986740    5659 generic.go:356] "Generic (PLEG): container finished" podID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerID="208cb05cb678e62705b2663440c26a5ccc690ce877d907c6a205f699b8c76ae1" exitCode=0
Nov 03 19:37:27.992023 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:27.991999    5659 generic.go:356] "Generic (PLEG): container finished" podID="583fd7cf27a9a1aaee6d847adb49df9a" containerID="32b1aa6939637b01a067414caad8da0bbeb9e2ddff9e9a2335313a37c6d6d60c" exitCode=0
Nov 03 19:37:28.004210 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.004182    5659 generic.go:356] "Generic (PLEG): container finished" podID="fc6cb168-a115-4918-a092-bc340accd868" containerID="368f7c54dbb72eb4fe5110fa39387632214bd73a6943b68bbd1be7fc7405e32b" exitCode=0
Nov 03 19:37:28.004210 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.004207    5659 generic.go:356] "Generic (PLEG): container finished" podID="fc6cb168-a115-4918-a092-bc340accd868" containerID="97f4800c37ebec47064220fc1e5d031e506a6290aaa469d85471ef25c1011e94" exitCode=0
Nov 03 19:37:28.004210 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.004216    5659 generic.go:356] "Generic (PLEG): container finished" podID="fc6cb168-a115-4918-a092-bc340accd868" containerID="487e3f941ff588cc5856d84658a2607130f05c224a1341c3c57bd5c4347cb350" exitCode=0
Nov 03 19:37:28.004463 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.004226    5659 generic.go:356] "Generic (PLEG): container finished" podID="fc6cb168-a115-4918-a092-bc340accd868" containerID="38d0f483ba10836fe1a13f0a313e63e52485fbb1bcaa8018ed7bd4ac1f3de4cb" exitCode=0
Nov 03 19:37:28.004463 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.004237    5659 generic.go:356] "Generic (PLEG): container finished" podID="fc6cb168-a115-4918-a092-bc340accd868" containerID="3b129ad0cafac257d8eabfd88d25b13adc9d6c0492c7ccca6110bb5a3de8b3d3" exitCode=0
Nov 03 19:37:28.004463 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.004245    5659 generic.go:356] "Generic (PLEG): container finished" podID="fc6cb168-a115-4918-a092-bc340accd868" containerID="3c9293b28b0ba902f2988b0e6b317b78e21e27c17515926d8e452ba7b18c7e29" exitCode=0
Nov 03 19:37:28.044884 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.044808    5659 manager.go:324] Recovery completed
Nov 03 19:37:28.061500 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:28.061452    5659 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 19:37:28.071170 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.071146    5659 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:37:28.071170 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.071163    5659 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:37:28.071346 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.071181    5659 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:37:28.071346 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.071317    5659 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 19:37:28.071346 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.071328    5659 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 19:37:28.071456 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.071348    5659 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 19:37:28.071456 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.071353    5659 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 19:37:28.071456 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.071359    5659 policy_none.go:50] "Start"
Nov 03 19:37:28.071456 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.071368    5659 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:37:28.071456 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.071377    5659 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:37:28.071612 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.071511    5659 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 19:37:28.071612 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.071519    5659 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 19:37:28.071612 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.071525    5659 policy_none.go:44] "Start"
Nov 03 19:37:28.076349 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.076313    5659 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:37:28.081572 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.081552    5659 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:37:28.082096 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:28.081583    5659 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:37:28.082168 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.082099    5659 server.go:87] "Starting device plugin registration server"
Nov 03 19:37:28.082389 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.082378    5659 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:37:28.082436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.082390    5659 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:37:28.082534 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.082515    5659 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:37:28.082671 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.082622    5659 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:37:28.082671 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.082670    5659 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:37:28.082820 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.082677    5659 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:37:28.082820 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.082729    5659 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:37:28.082653567Z","UUID":"dd2a6bd5-375f-4ac2-a0a8-3baa0f0b8e7d","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:37:28.083971 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.083951    5659 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:37:28.084045 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.083978    5659 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:37:28.084645 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:28.084624    5659 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:37:28.182885 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.182847    5659 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:37:28.185021 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.184995    5659 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:37:28.185142 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.185038    5659 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:37:28.185142 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.185049    5659 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-249.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:37:28.185142 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.185116    5659 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.198685 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.198659    5659 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.198823 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.198745    5659 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.261978 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.261900    5659 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"]
Nov 03 19:37:28.262590 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.262554    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal" event={"ID":"583fd7cf27a9a1aaee6d847adb49df9a","Type":"ContainerStarted","Data":"cb7933fdab6412a6232106d1f9c7cc4486c5936dd2f2fedf10d6f71b7ff797b1"}
Nov 03 19:37:28.262656 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.262598    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal" event={"ID":"583fd7cf27a9a1aaee6d847adb49df9a","Type":"ContainerDied","Data":"32b1aa6939637b01a067414caad8da0bbeb9e2ddff9e9a2335313a37c6d6d60c"}
Nov 03 19:37:28.262656 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.262618    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal" event={"ID":"583fd7cf27a9a1aaee6d847adb49df9a","Type":"ContainerStarted","Data":"e4573b6c07d46d668269dd08759155a50b743f1ab4fae0d8f764e63052809f55"}
Nov 03 19:37:28.262712 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.262698    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal" event={"ID":"465e529db144367f617f146802de7125","Type":"ContainerStarted","Data":"cad7672a21a325988a408255f0c5367ae045c4b500f71d954a808eaecca83607"}
Nov 03 19:37:28.262746 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.262710    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal" event={"ID":"465e529db144367f617f146802de7125","Type":"ContainerStarted","Data":"fa4b784afcde48d129fed5739bd18e1912366eb56456f5af792846354f6c0389"}
Nov 03 19:37:28.262746 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.262713    5659 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.262804 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.262774    5659 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.276689 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.276657    5659 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:37:28.276822 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:28.276715    5659 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.276822 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.276657    5659 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:37:28.276822 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:28.276792    5659 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.349162 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.349070    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/465e529db144367f617f146802de7125-config\") pod \"kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"465e529db144367f617f146802de7125\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.349162 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.349109    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/583fd7cf27a9a1aaee6d847adb49df9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"583fd7cf27a9a1aaee6d847adb49df9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.349162 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.349128    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/583fd7cf27a9a1aaee6d847adb49df9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"583fd7cf27a9a1aaee6d847adb49df9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.449694 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.449660    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/465e529db144367f617f146802de7125-config\") pod \"kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"465e529db144367f617f146802de7125\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.449790 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.449700    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/583fd7cf27a9a1aaee6d847adb49df9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"583fd7cf27a9a1aaee6d847adb49df9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.449790 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.449728    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/583fd7cf27a9a1aaee6d847adb49df9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"583fd7cf27a9a1aaee6d847adb49df9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.449790 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.449769    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/465e529db144367f617f146802de7125-config\") pod \"kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"465e529db144367f617f146802de7125\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.449790 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.449777    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/583fd7cf27a9a1aaee6d847adb49df9a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"583fd7cf27a9a1aaee6d847adb49df9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.449790 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.449780    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/583fd7cf27a9a1aaee6d847adb49df9a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" (UID: \"583fd7cf27a9a1aaee6d847adb49df9a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:28.916328 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.916292    5659 apiserver.go:52] "Watching apiserver"
Nov 03 19:37:28.929721 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.929686    5659 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:37:28.930422 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.930400    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-6q6gg","openshift-ingress-canary/ingress-canary-pqv25","kube-system/global-pull-secret-syncer-hc967","openshift-cluster-node-tuning-operator/tuned-vbcxg","openshift-dns/dns-default-5xsml","openshift-monitoring/telemeter-client-7755d4f754-nrkkn","openshift-console/console-7d6b976d69-rjgm5","openshift-insights/insights-runtime-extractor-kgm4l","openshift-console/console-5d4cc48b5f-9g4kc","openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr","openshift-multus/network-metrics-daemon-bpk6k","openshift-network-diagnostics/network-check-target-vcw9t","kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal","openshift-console/console-67d4ffd4f9-rc92g","openshift-console/console-6cdcc6bb85-48j6f","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-additional-cni-plugins-bt6hc","openshift-monitoring/node-exporter-9wvlh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws","openshift-image-registry/node-ca-mmxqm","openshift-ovn-kubernetes/ovnkube-node-v2dj5","openshift-dns/node-resolver-5ktfb","openshift-multus/multus-k9k97","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal","openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx","openshift-network-operator/iptables-alerter-5qqvk"]
Nov 03 19:37:28.936242 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.936213    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ldllm\""
Nov 03 19:37:28.936402 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.936294    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bkd9f\""
Nov 03 19:37:28.936402 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.936377    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lrbp4\""
Nov 03 19:37:28.937012 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.936990    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbkg2\""
Nov 03 19:37:28.937126 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.937089    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:37:28.937126 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.937102    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:37:28.937234 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.937131    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:37:28.937283 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.937249    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:37:28.937497 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.937463    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:37:28.937606 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.937465    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:37:28.937606 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.937505    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:37:28.937606 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.937508    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:37:28.937787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.937531    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:37:28.937787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.937631    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:37:28.938423 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.938391    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:37:28.938423 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.938405    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rs44g\""
Nov 03 19:37:28.938580 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.938465    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:37:28.938580 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.938502    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:37:28.938580 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.938509    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:37:28.938580 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.938556    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:37:28.938580 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.938565    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:37:28.938793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.938585    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:37:28.938850 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.938810    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:37:28.938850 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.938820    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vlmgx\""
Nov 03 19:37:28.938850 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.938846    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:37:28.939762 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.939742    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:37:28.939845 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.939775    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:37:28.939845 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.939777    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndfpg\""
Nov 03 19:37:28.939845 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.939795    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:37:28.939845 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.939796    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:37:28.940025 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.939853    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:37:28.940025 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.939864    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:37:28.940025 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.939913    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xv4gr\""
Nov 03 19:37:28.940025 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.939956    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:37:28.940184 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.940165    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhpjg\""
Nov 03 19:37:28.940708 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.940691    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzp4r\""
Nov 03 19:37:28.940816 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.940802    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:37:28.941064 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.941049    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx8mk\""
Nov 03 19:37:28.941460 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.941443    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5qm8x\""
Nov 03 19:37:28.941943 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.941928    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:37:28.942043 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.941948    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qg9jj\""
Nov 03 19:37:28.942581 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.942565    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 19:37:28.942813 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.942794    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 19:37:28.942915 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.942864    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-779gt\""
Nov 03 19:37:28.943455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.943434    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgjzc\""
Nov 03 19:37:28.943591 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.943476    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b2f85\""
Nov 03 19:37:28.943591 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.943524    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:37:28.945322 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.945298    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:37:28.945574 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.945555    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:37:28.952197 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.952172    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 19:37:28.952289 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.952268    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 19:37:28.952521 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.952503    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:37:28.952521 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.952512    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:37:28.952521 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.952506    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:37:28.952678 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.952586    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 19:37:28.952678 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.952647    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:37:28.952737 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:28.952699    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:37:29.030104 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.030069    5659 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:29.030262 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.030215    5659 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:29.032181 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.032165    5659 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:37:29.036082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036062    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 19:37:29.036082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036061    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 19:37:29.036248 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036161    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 19:37:29.036248 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036186    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 19:37:29.036336 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036265    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 19:37:29.036336 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036290    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:37:29.036427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036347    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fqsxm\""
Nov 03 19:37:29.036427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036349    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 19:37:29.036427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036384    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 19:37:29.036581 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036568    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 19:37:29.036636 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036593    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 19:37:29.036636 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036598    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 19:37:29.036636 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036611    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 19:37:29.036926 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.036909    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mkqv6\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.037626    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.037629    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.037670    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.037718    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.037905    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lhxcg\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.037969    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.037988    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038062    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038104    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038126    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038129    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.037910    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038210    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038270    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1kr3opstr93m9\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038338    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-w7r7w\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038397    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zljmq\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038449    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4nue8o8t41bmn\""
Nov 03 19:37:29.038589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038511    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 19:37:29.039160 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038848    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 19:37:29.039160 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038886    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:37:29.039160 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038911    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 19:37:29.039160 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038914    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:37:29.039160 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.038944    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 19:37:29.039360 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.039227    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:37:29.039360 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.039231    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 19:37:29.039360 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.039248    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 19:37:29.039360 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.039262    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 19:37:29.040615 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.040600    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 19:37:29.048915 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.048897    5659 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:37:29.048996 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:29.048948    5659 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:29.048996 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.048897    5659 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:37:29.049147 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:29.049027    5659 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal"
Nov 03 19:37:29.051103 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.051074    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 19:37:29.052358 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.052335    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 19:37:29.052445 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.052392    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 19:37:29.052627 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.052609    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 19:37:29.052900 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.052879    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-cni-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.052971 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.052928    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.052971 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.052957    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae317c95-3690-4603-a062-652384d87f55-kubelet-config\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:37:29.053063 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.052991    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-serving-cert\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.053063 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053024    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d03e8db-60e2-4844-94c3-78f02bc19100-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:29.053063 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053053    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b6a3864-3172-40f0-9ab4-7327e2bf461c-tmp\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.053194 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053101    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a0edbce3-cc93-410d-b82e-27e13cb6412c-metrics-server-audit-profiles\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.053194 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053144    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6k67\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-kube-api-access-d6k67\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.053194 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053156    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b6a3864-3172-40f0-9ab4-7327e2bf461c-tmp\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.053194 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053179    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dd8ccdd-1987-4200-8963-983dc4fd445d-cni-binary-copy\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.053366 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053227    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-oauth-config\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.053366 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053260    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146c892e-d6ef-4062-94c4-6dae80875952-metrics-client-ca\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.053366 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053287    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysconfig\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.053366 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053322    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.053366 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053354    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxm9f\" (UniqueName: \"kubernetes.io/projected/fc6cb168-a115-4918-a092-bc340accd868-kube-api-access-zxm9f\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.053597 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053383    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.053597 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053404    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dd8ccdd-1987-4200-8963-983dc4fd445d-cni-binary-copy\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.053597 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053411    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99464aea-b429-4992-b4dd-ad48c7d99633-tmp\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.053597 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053438    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d03e8db-60e2-4844-94c3-78f02bc19100-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:29.053781 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053761    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-serving-cert\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.054016 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053994    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/146c892e-d6ef-4062-94c4-6dae80875952-metrics-client-ca\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.054104 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054086    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a0edbce3-cc93-410d-b82e-27e13cb6412c-metrics-server-audit-profiles\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.054150 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054088    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99464aea-b429-4992-b4dd-ad48c7d99633-tmp\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.054193 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054159    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-secret-telemeter-client\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.054244 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054194    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d03e8db-60e2-4844-94c3-78f02bc19100-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:29.054244 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054224    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-federate-client-tls\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.054343 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054255    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e50201a6-9ca6-4ff0-9e55-cc324745331e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:29.054343 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054284    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-trusted-ca-bundle\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.054343 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054286    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d03e8db-60e2-4844-94c3-78f02bc19100-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:29.054343 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054310    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e79d1e48-bb87-4a1b-9b88-95a1551e959c-hosts-file\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:37:29.054530 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054345    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.054530 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054376    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-telemeter-client-tls\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.054530 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054410    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.054530 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054437    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7knzb\" (UniqueName: \"kubernetes.io/projected/99464aea-b429-4992-b4dd-ad48c7d99633-kube-api-access-7knzb\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.054530 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054467    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c43e3b1f-9e9c-4768-8c45-e779762180bd-iptables-alerter-script\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:37:29.054530 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054515    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-config\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.054811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054549    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6n4\" (UniqueName: \"kubernetes.io/projected/4af39c8b-d06b-4569-bc27-ca478cc4852c-kube-api-access-wn6n4\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.054811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054652    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/674ad509-03fd-4f35-bc70-3a3602457de9-config-volume\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:29.054811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054722    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-oauth-serving-cert\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.054811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054736    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.054811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.053753    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d03e8db-60e2-4844-94c3-78f02bc19100-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:29.055082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054829    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-telemeter-client-tls\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.055082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054921    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-oauth-config\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.055082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054960    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/674ad509-03fd-4f35-bc70-3a3602457de9-config-volume\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:29.055082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.054967    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-oauth-serving-cert\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.055082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055011    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-config\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.055082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055011    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e50201a6-9ca6-4ff0-9e55-cc324745331e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:29.055082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055059    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-federate-client-tls\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.055394 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055097    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.055394 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055199    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.055394 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055243    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-secret-telemeter-client\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.055394 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055286    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-conf-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.055589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055534    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d03e8db-60e2-4844-94c3-78f02bc19100-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:29.055589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055577    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-oauth-config\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.055731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055712    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-oauth-config\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.055731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055720    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-metrics-client-ca\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.055833 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055785    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ad627830-62cd-4f5b-bb63-76b5944b91ff-serviceca\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:37:29.055880 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055858    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.055935 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055855    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c43e3b1f-9e9c-4768-8c45-e779762180bd-iptables-alerter-script\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:37:29.055935 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055922    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28xl5\" (UniqueName: \"kubernetes.io/projected/e79d1e48-bb87-4a1b-9b88-95a1551e959c-kube-api-access-28xl5\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:37:29.056034 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055951    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-metrics-client-ca\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.056034 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055994    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-trusted-ca-bundle\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.056034 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.055997    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.056245 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056222    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-serving-cert\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.056341 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056273    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-sys\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.056667 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056648    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.056722 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056687    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:37:29.056773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056728    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbk8q\" (UniqueName: \"kubernetes.io/projected/674ad509-03fd-4f35-bc70-3a3602457de9-kube-api-access-nbk8q\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:29.056773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056755    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.056882 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056782    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-service-ca\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.056882 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056783    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-serving-cert\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.056882 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056808    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-oauth-serving-cert\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.056882 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056833    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp2b4\" (UniqueName: \"kubernetes.io/projected/146c892e-d6ef-4062-94c4-6dae80875952-kube-api-access-vp2b4\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.056882 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056859    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-systemd-units\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.057118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056885    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-secret-metrics-server-tls\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.057118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056912    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plbb9\" (UniqueName: \"kubernetes.io/projected/a0edbce3-cc93-410d-b82e-27e13cb6412c-kube-api-access-plbb9\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.057118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056937    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-trusted-ca-bundle\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.057118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056961    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-socket-dir-parent\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.057118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056972    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658b22ed-1324-4c81-b112-f051fdd3baa4-metrics-certs\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:37:29.057118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.056986    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-multus-certs\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.057118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057018    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.057118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057062    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ad627830-62cd-4f5b-bb63-76b5944b91ff-serviceca\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:37:29.057118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057070    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48pr9\" (UniqueName: \"kubernetes.io/projected/93b47942-ccbf-43cd-8722-c21038f906f3-kube-api-access-48pr9\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057125    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-etc-selinux\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057154    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-run\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057198    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-node-log\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057209    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-service-ca\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057224    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-log-socket\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057249    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-oauth-serving-cert\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057252    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-console-config\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057276    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-secret-metrics-server-tls\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057375    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057412    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057476    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-host\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057541    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-trusted-ca-bundle\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057577    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-device-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.057611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057616    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-etc-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.058014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057654    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.058014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057669    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.058014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057672    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.058014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057802    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-kubelet\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.058014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057824    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.058014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057830    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.058014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057843    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqjz\" (UniqueName: \"kubernetes.io/projected/658b22ed-1324-4c81-b112-f051fdd3baa4-kube-api-access-lbqjz\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:37:29.058014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057858    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-console-config\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.058014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057904    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-console-config\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.058014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057959    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-config\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.058014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057971    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93b47942-ccbf-43cd-8722-c21038f906f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.058014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.057987    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-tls\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.058014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058004    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-wtmp\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.058468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058020    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e50201a6-9ca6-4ff0-9e55-cc324745331e-data-volume\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:29.058468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058062    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-service-ca\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.058468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058135    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysctl-conf\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.058468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058151    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e50201a6-9ca6-4ff0-9e55-cc324745331e-data-volume\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:29.058468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058168    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8wnj\" (UniqueName: \"kubernetes.io/projected/4d03e8db-60e2-4844-94c3-78f02bc19100-kube-api-access-m8wnj\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:29.058468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058217    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-oauth-serving-cert\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.058468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058229    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-tls\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.058468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058273    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-socket-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.058468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058323    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-service-ca\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.058468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058329    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e79d1e48-bb87-4a1b-9b88-95a1551e959c-tmp-dir\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:37:29.058468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058406    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e79d1e48-bb87-4a1b-9b88-95a1551e959c-tmp-dir\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:37:29.058468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058426    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-hostroot\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.058805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058476    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-oauth-serving-cert\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.058805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058473    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-root\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.058805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058524    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-console-config\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.058805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058534    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d6f9087-55e4-4e30-905a-3a1493336ded-ovn-node-metrics-cert\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.058805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058563    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-config\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.058805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058575    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j68r2\" (UniqueName: \"kubernetes.io/projected/ad627830-62cd-4f5b-bb63-76b5944b91ff-kube-api-access-j68r2\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:37:29.058805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058626    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8236a6c-f408-4996-906d-a73a8cb0069d-tmp\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.058805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058704    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d6f9087-55e4-4e30-905a-3a1493336ded-ovn-node-metrics-cert\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.058805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058738    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-cnibin\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.058805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058768    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8236a6c-f408-4996-906d-a73a8cb0069d-tmp\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.058805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058776    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.058805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058798    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ndj\" (UniqueName: \"kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj\") pod \"network-check-target-vcw9t\" (UID: \"124e9f93-f05c-4eee-8054-ac67754550c3\") " pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058814    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-tuned\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058830    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058847    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-oauth-config\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058870    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttmj2\" (UniqueName: \"kubernetes.io/projected/c43e3b1f-9e9c-4768-8c45-e779762180bd-kube-api-access-ttmj2\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058894    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-daemon-config\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058901    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058914    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-tuned\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058918    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-trusted-ca-bundle\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058967    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmfq\" (UniqueName: \"kubernetes.io/projected/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-kube-api-access-xrmfq\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.058995    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-ovnkube-script-lib\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059019    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059029    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-oauth-config\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059046    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-serving-cert\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059071    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-service-ca\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059094    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d-agent-certs\") pod \"konnectivity-agent-6q6gg\" (UID: \"16b10ae2-69df-49f1-b0b3-77c98ee4cc3d\") " pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059084    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059125    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-daemon-config\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059169    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-trusted-ca-bundle\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059175    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4rw\" (UniqueName: \"kubernetes.io/projected/51cdd330-78ed-418e-8bdb-1f080d7c2b11-kube-api-access-fs4rw\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059222    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-textfile\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059249    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-ovnkube-script-lib\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059276    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-env-overrides\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059309    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0edbce3-cc93-410d-b82e-27e13cb6412c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059328    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-service-ca\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059337    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.059340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059331    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-serving-cert\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059356    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059384    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d-agent-certs\") pod \"konnectivity-agent-6q6gg\" (UID: \"16b10ae2-69df-49f1-b0b3-77c98ee4cc3d\") " pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059333    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-textfile\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059363    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-service-ca\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059450    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-registration-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059473    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/674ad509-03fd-4f35-bc70-3a3602457de9-metrics-tls\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059494    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-env-overrides\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059521    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-cnibin\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059529    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0edbce3-cc93-410d-b82e-27e13cb6412c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059537    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-etc-kubernetes\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059558    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51cdd330-78ed-418e-8bdb-1f080d7c2b11-tmp\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059563    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-service-ca\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059590    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-systemd\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059598    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059619    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51cdd330-78ed-418e-8bdb-1f080d7c2b11-tmp\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059642    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-oauth-config\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059676    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-web-config\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059689    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/674ad509-03fd-4f35-bc70-3a3602457de9-metrics-tls\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059706    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysctl-d\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059733    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-serving-cert\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059760    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-cni-multus\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059778    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-oauth-config\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059788    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74569270-9489-4c9f-b51e-d819081baa5d-cert\") pod \"ingress-canary-pqv25\" (UID: \"74569270-9489-4c9f-b51e-d819081baa5d\") " pod="openshift-ingress-canary/ingress-canary-pqv25"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059817    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059824    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-web-config\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059857    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-systemd\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059893    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-serving-cert\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059924    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config-out\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059952    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74569270-9489-4c9f-b51e-d819081baa5d-cert\") pod \"ingress-canary-pqv25\" (UID: \"74569270-9489-4c9f-b51e-d819081baa5d\") " pod="openshift-ingress-canary/ingress-canary-pqv25"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059955    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ssv\" (UniqueName: \"kubernetes.io/projected/2b6a3864-3172-40f0-9ab4-7327e2bf461c-kube-api-access-v8ssv\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059969    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059997    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config-out\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.059993    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-serving-certs-ca-bundle\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060070    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e50201a6-9ca6-4ff0-9e55-cc324745331e-crio-socket\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060093    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-client-ca-bundle\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060108    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060140    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-netns\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060161    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-serving-certs-ca-bundle\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060176    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nngxb\" (UniqueName: \"kubernetes.io/projected/4dd8ccdd-1987-4200-8963-983dc4fd445d-kube-api-access-nngxb\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060198    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctw8s\" (UniqueName: \"kubernetes.io/projected/74569270-9489-4c9f-b51e-d819081baa5d-kube-api-access-ctw8s\") pod \"ingress-canary-pqv25\" (UID: \"74569270-9489-4c9f-b51e-d819081baa5d\") " pod="openshift-ingress-canary/ingress-canary-pqv25"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060224    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-kubernetes\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060327    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-run-netns\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.060327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060348    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060368    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060383    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-client-ca-bundle\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060391    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060421    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a0edbce3-cc93-410d-b82e-27e13cb6412c-audit-log\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060463    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d-konnectivity-ca\") pod \"konnectivity-agent-6q6gg\" (UID: \"16b10ae2-69df-49f1-b0b3-77c98ee4cc3d\") " pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060515    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a0edbce3-cc93-410d-b82e-27e13cb6412c-audit-log\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060566    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060591    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-cni-bin\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060612    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060644    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-lib-modules\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060661    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060675    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16b10ae2-69df-49f1-b0b3-77c98ee4cc3d-konnectivity-ca\") pod \"konnectivity-agent-6q6gg\" (UID: \"16b10ae2-69df-49f1-b0b3-77c98ee4cc3d\") " pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060679    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e50201a6-9ca6-4ff0-9e55-cc324745331e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060697    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-oauth-serving-cert\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060803    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-system-cni-dir\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060837    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060872    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-oauth-serving-cert\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060879    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e50201a6-9ca6-4ff0-9e55-cc324745331e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.060963    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c43e3b1f-9e9c-4768-8c45-e779762180bd-host-slash\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061001    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-cni-bin\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061030    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-ovnkube-config\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061081    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-system-cni-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061115    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-k8s-cni-cncf-io\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061156    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-sys-fs\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061184    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061186    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d6f9087-55e4-4e30-905a-3a1493336ded-ovnkube-config\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061210    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061306    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-sys\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061357    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-var-lib-kubelet\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061380    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-slash\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061393    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061401    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad627830-62cd-4f5b-bb63-76b5944b91ff-host\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061427    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj9m7\" (UniqueName: \"kubernetes.io/projected/c8236a6c-f408-4996-906d-a73a8cb0069d-kube-api-access-sj9m7\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061459    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-secret-metrics-server-client-certs\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061511    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061539    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-var-lib-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061564    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-ovn\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061594    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674ad509-03fd-4f35-bc70-3a3602457de9-tmp-dir\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061638    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-kubelet\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061675    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thm8g\" (UniqueName: \"kubernetes.io/projected/2d6f9087-55e4-4e30-905a-3a1493336ded-kube-api-access-thm8g\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061700    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674ad509-03fd-4f35-bc70-3a3602457de9-tmp-dir\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061701    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfpsh\" (UniqueName: \"kubernetes.io/projected/e50201a6-9ca6-4ff0-9e55-cc324745331e-kube-api-access-dfpsh\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061712    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a0edbce3-cc93-410d-b82e-27e13cb6412c-secret-metrics-server-client-certs\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061730    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061762    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae317c95-3690-4603-a062-652384d87f55-dbus\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061766    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061770    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061791    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-tmp\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.061787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061810    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.063111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061831    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-modprobe-d\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.063111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061867    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-cni-netd\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.063111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061869    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-tmp\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.063111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061883    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-cni-binary-copy\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.063111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.061984    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.063111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.062007    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:37:29.063111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.062022    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.063111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.062138    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93b47942-ccbf-43cd-8722-c21038f906f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.063111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.062186    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.063111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.062207    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc6cb168-a115-4918-a092-bc340accd868-cni-binary-copy\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.063111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.062404    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae317c95-3690-4603-a062-652384d87f55-original-pull-secret\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:37:29.064524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.064508    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6n4\" (UniqueName: \"kubernetes.io/projected/4af39c8b-d06b-4569-bc27-ca478cc4852c-kube-api-access-wn6n4\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.064524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.064514    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxm9f\" (UniqueName: \"kubernetes.io/projected/fc6cb168-a115-4918-a092-bc340accd868-kube-api-access-zxm9f\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.064727 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.064713    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6k67\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-kube-api-access-d6k67\") pod \"prometheus-k8s-0\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:29.065096 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.065082    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7knzb\" (UniqueName: \"kubernetes.io/projected/99464aea-b429-4992-b4dd-ad48c7d99633-kube-api-access-7knzb\") pod \"console-67d4ffd4f9-rc92g\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") " pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:29.065274 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.065259    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28xl5\" (UniqueName: \"kubernetes.io/projected/e79d1e48-bb87-4a1b-9b88-95a1551e959c-kube-api-access-28xl5\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:37:29.067947 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.067927    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqjz\" (UniqueName: \"kubernetes.io/projected/658b22ed-1324-4c81-b112-f051fdd3baa4-kube-api-access-lbqjz\") pod \"network-metrics-daemon-bpk6k\" (UID: \"658b22ed-1324-4c81-b112-f051fdd3baa4\") " pod="openshift-multus/network-metrics-daemon-bpk6k"
Nov 03 19:37:29.068201 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.068184    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp2b4\" (UniqueName: \"kubernetes.io/projected/146c892e-d6ef-4062-94c4-6dae80875952-kube-api-access-vp2b4\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.068280 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.068246    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbk8q\" (UniqueName: \"kubernetes.io/projected/674ad509-03fd-4f35-bc70-3a3602457de9-kube-api-access-nbk8q\") pod \"dns-default-5xsml\" (UID: \"674ad509-03fd-4f35-bc70-3a3602457de9\") " pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:29.068356 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.068340    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plbb9\" (UniqueName: \"kubernetes.io/projected/a0edbce3-cc93-410d-b82e-27e13cb6412c-kube-api-access-plbb9\") pod \"metrics-server-5fc5d9f9bb-s5bxr\" (UID: \"a0edbce3-cc93-410d-b82e-27e13cb6412c\") " pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:29.068511 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.068495    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48pr9\" (UniqueName: \"kubernetes.io/projected/93b47942-ccbf-43cd-8722-c21038f906f3-kube-api-access-48pr9\") pod \"telemeter-client-7755d4f754-nrkkn\" (UID: \"93b47942-ccbf-43cd-8722-c21038f906f3\") " pod="openshift-monitoring/telemeter-client-7755d4f754-nrkkn"
Nov 03 19:37:29.073082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.073061    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttmj2\" (UniqueName: \"kubernetes.io/projected/c43e3b1f-9e9c-4768-8c45-e779762180bd-kube-api-access-ttmj2\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:37:29.073756 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.073549    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8wnj\" (UniqueName: \"kubernetes.io/projected/4d03e8db-60e2-4844-94c3-78f02bc19100-kube-api-access-m8wnj\") pod \"openshift-state-metrics-7bd7489957-v6ptx\" (UID: \"4d03e8db-60e2-4844-94c3-78f02bc19100\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-v6ptx"
Nov 03 19:37:29.073756 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.073632    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j68r2\" (UniqueName: \"kubernetes.io/projected/ad627830-62cd-4f5b-bb63-76b5944b91ff-kube-api-access-j68r2\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:37:29.073756 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.073749    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4ndj\" (UniqueName: \"kubernetes.io/projected/124e9f93-f05c-4eee-8054-ac67754550c3-kube-api-access-l4ndj\") pod \"network-check-target-vcw9t\" (UID: \"124e9f93-f05c-4eee-8054-ac67754550c3\") " pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:37:29.074541 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.074469    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nngxb\" (UniqueName: \"kubernetes.io/projected/4dd8ccdd-1987-4200-8963-983dc4fd445d-kube-api-access-nngxb\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.074878 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.074861    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ssv\" (UniqueName: \"kubernetes.io/projected/2b6a3864-3172-40f0-9ab4-7327e2bf461c-kube-api-access-v8ssv\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.074936 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.074875    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4rw\" (UniqueName: \"kubernetes.io/projected/51cdd330-78ed-418e-8bdb-1f080d7c2b11-kube-api-access-fs4rw\") pod \"console-5d4cc48b5f-9g4kc\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") " pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:29.074936 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.074915    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctw8s\" (UniqueName: \"kubernetes.io/projected/74569270-9489-4c9f-b51e-d819081baa5d-kube-api-access-ctw8s\") pod \"ingress-canary-pqv25\" (UID: \"74569270-9489-4c9f-b51e-d819081baa5d\") " pod="openshift-ingress-canary/ingress-canary-pqv25"
Nov 03 19:37:29.075049 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.075033    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmfq\" (UniqueName: \"kubernetes.io/projected/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-kube-api-access-xrmfq\") pod \"console-7d6b976d69-rjgm5\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") " pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:29.078796 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.078779    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj9m7\" (UniqueName: \"kubernetes.io/projected/c8236a6c-f408-4996-906d-a73a8cb0069d-kube-api-access-sj9m7\") pod \"console-6cdcc6bb85-48j6f\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") " pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:29.079398 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.079383    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thm8g\" (UniqueName: \"kubernetes.io/projected/2d6f9087-55e4-4e30-905a-3a1493336ded-kube-api-access-thm8g\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.081187 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.081172    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfpsh\" (UniqueName: \"kubernetes.io/projected/e50201a6-9ca6-4ff0-9e55-cc324745331e-kube-api-access-dfpsh\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:29.162681 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162647    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-sys\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.162681 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162681    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-systemd-units\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162703    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-socket-dir-parent\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162725    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-multus-certs\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162753    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-sys\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162762    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-systemd-units\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162769    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-multus-certs\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162763    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-etc-selinux\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162813    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-run\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162822    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-socket-dir-parent\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162813    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-etc-selinux\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162833    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-node-log\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162840    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-run\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162852    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-log-socket\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162862    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-node-log\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162869    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-host\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162894    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-device-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.162901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162910    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-etc-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162941    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-etc-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162952    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-device-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162959    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-log-socket\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162952    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-host\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162971    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-kubelet\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.162997    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-wtmp\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163015    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysctl-conf\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163035    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-kubelet\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163041    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-socket-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163063    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-node-exporter-wtmp\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163072    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-hostroot\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163093    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-socket-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163097    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-root\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163124    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-cnibin\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163137    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-hostroot\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163168    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysctl-conf\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163169    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-registration-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163196    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-cnibin\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163205    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/146c892e-d6ef-4062-94c4-6dae80875952-root\") pod \"node-exporter-9wvlh\" (UID: \"146c892e-d6ef-4062-94c4-6dae80875952\") " pod="openshift-monitoring/node-exporter-9wvlh"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163211    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-etc-kubernetes\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163243    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-systemd\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163247    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-etc-kubernetes\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163272    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysctl-d\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163314    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-cni-multus\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163346    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-cnibin\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163351    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-systemd\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163382    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e50201a6-9ca6-4ff0-9e55-cc324745331e-crio-socket\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163352    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-systemd\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163383    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-cni-multus\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163386    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-systemd\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163174    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-cnibin\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163413    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-netns\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163437    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e50201a6-9ca6-4ff0-9e55-cc324745331e-crio-socket\") pod \"insights-runtime-extractor-kgm4l\" (UID: \"e50201a6-9ca6-4ff0-9e55-cc324745331e\") " pod="openshift-insights/insights-runtime-extractor-kgm4l"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163446    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-kubernetes\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163465    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-netns\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163474    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-run-netns\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163518    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-run-netns\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163521    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163552    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-cni-bin\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163558    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163474    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysctl-d\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163585    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163525    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-kubernetes\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163598    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-var-lib-cni-bin\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163602    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-lib-modules\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163632    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163652    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163652    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-system-cni-dir\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163682    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc6cb168-a115-4918-a092-bc340accd868-system-cni-dir\") pod \"multus-additional-cni-plugins-bt6hc\" (UID: \"fc6cb168-a115-4918-a092-bc340accd868\") " pod="openshift-multus/multus-additional-cni-plugins-bt6hc"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163668    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-lib-modules\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163692    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c43e3b1f-9e9c-4768-8c45-e779762180bd-host-slash\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163697    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.163688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163710    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-cni-bin\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163208    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-registration-dir\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163732    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-system-cni-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163767    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-cni-bin\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163769    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c43e3b1f-9e9c-4768-8c45-e779762180bd-host-slash\") pod \"iptables-alerter-5qqvk\" (UID: \"c43e3b1f-9e9c-4768-8c45-e779762180bd\") " pod="openshift-network-operator/iptables-alerter-5qqvk"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163779    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-k8s-cni-cncf-io\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163789    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-system-cni-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163808    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-sys-fs\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163823    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163850    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-sys\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163876    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-var-lib-kubelet\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163882    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163885    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4af39c8b-d06b-4569-bc27-ca478cc4852c-sys-fs\") pod \"aws-ebs-csi-driver-node-6x6ws\" (UID: \"4af39c8b-d06b-4569-bc27-ca478cc4852c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x6ws"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163900    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-slash\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163925    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-host-run-k8s-cni-cncf-io\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163926    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad627830-62cd-4f5b-bb63-76b5944b91ff-host\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163941    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-var-lib-kubelet\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163953    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-sys\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.163984    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad627830-62cd-4f5b-bb63-76b5944b91ff-host\") pod \"node-ca-mmxqm\" (UID: \"ad627830-62cd-4f5b-bb63-76b5944b91ff\") " pod="openshift-image-registry/node-ca-mmxqm"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164003    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-slash\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164020    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-var-lib-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164039    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-var-lib-openvswitch\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164042    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-ovn\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164064    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-run-ovn\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164070    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-kubelet\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164098    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae317c95-3690-4603-a062-652384d87f55-dbus\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164114    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-modprobe-d\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164129    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-cni-netd\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164145    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-cni-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164153    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae317c95-3690-4603-a062-652384d87f55-dbus\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164171    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae317c95-3690-4603-a062-652384d87f55-kubelet-config\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164183    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-cni-netd\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164200    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysconfig\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164106    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-kubelet\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164207    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-cni-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164213    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-modprobe-d\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164217    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae317c95-3690-4603-a062-652384d87f55-kubelet-config\") pod \"global-pull-secret-syncer-hc967\" (UID: \"ae317c95-3690-4603-a062-652384d87f55\") " pod="kube-system/global-pull-secret-syncer-hc967"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164241    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164261    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b6a3864-3172-40f0-9ab4-7327e2bf461c-etc-sysconfig\") pod \"tuned-vbcxg\" (UID: \"2b6a3864-3172-40f0-9ab4-7327e2bf461c\") " pod="openshift-cluster-node-tuning-operator/tuned-vbcxg"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164278    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e79d1e48-bb87-4a1b-9b88-95a1551e959c-hosts-file\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164305    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6f9087-55e4-4e30-905a-3a1493336ded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2dj5\" (UID: \"2d6f9087-55e4-4e30-905a-3a1493336ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164339    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-conf-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164374    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e79d1e48-bb87-4a1b-9b88-95a1551e959c-hosts-file\") pod \"node-resolver-5ktfb\" (UID: \"e79d1e48-bb87-4a1b-9b88-95a1551e959c\") " pod="openshift-dns/node-resolver-5ktfb"
Nov 03 19:37:29.166175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.164389    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dd8ccdd-1987-4200-8963-983dc4fd445d-multus-conf-dir\") pod \"multus-k9k97\" (UID: \"4dd8ccdd-1987-4200-8963-983dc4fd445d\") " pod="openshift-multus/multus-k9k97"
Nov 03 19:37:29.228722 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.228690    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d4cc48b5f-9g4kc"]
Nov 03 19:37:29.302509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.302461    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:29.428686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:29.428651    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:31.084602 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:31.084564    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5xsml"
Nov 03 19:37:31.199601 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:31.199562    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:31.327348 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:31.327320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:32.703678 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:32.703637    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:32.926027 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:32.925993    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:33.062152 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:33.062118    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:33.486196 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:33.486164    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:33.504318 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:33.504289    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:34.186291 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:34.186258    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:34.293385 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:34.293352    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:34.452552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:34.452472    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:36.450181 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:36.450134    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:37:37.424511 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:37.424459    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:37.495016 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:37.494983    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:39.209494 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:39.209396    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:39.209864 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:39.209702    5659 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:37:40.711701 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:40.711670    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:40.836542 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:40.836506    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:42.709282 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:42.709246    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:42.712992 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:42.712964    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:37:42.935263 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:42.935220    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:43.062503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:43.062403    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:45.215726 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:45.215696    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:37:48.510184 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:48.510149    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5fc5d9f9bb-s5bxr"
Nov 03 19:37:49.414515 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:49.414465    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:49.538521 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:49.538466    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:51.224624 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:51.224601    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2dj5"
Nov 03 19:37:52.049646 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.049612    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b6799d47f-t448x"]
Nov 03 19:37:52.057589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.057566    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.066635 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.066610    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b6799d47f-t448x"]
Nov 03 19:37:52.072495 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.072458    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da47f42e-3aa5-4f3c-9b56-549760b8bab3-tmp\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.072600 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.072557    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da47f42e-3aa5-4f3c-9b56-549760b8bab3-trusted-ca-bundle\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.072600 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.072585    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da47f42e-3aa5-4f3c-9b56-549760b8bab3-console-serving-cert\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.072699 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.072604    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da47f42e-3aa5-4f3c-9b56-549760b8bab3-console-oauth-config\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.072739 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.072707    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcbvh\" (UniqueName: \"kubernetes.io/projected/da47f42e-3aa5-4f3c-9b56-549760b8bab3-kube-api-access-kcbvh\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.072739 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.072733    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da47f42e-3aa5-4f3c-9b56-549760b8bab3-service-ca\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.072829 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.072753    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da47f42e-3aa5-4f3c-9b56-549760b8bab3-oauth-serving-cert\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.072829 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.072775    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da47f42e-3aa5-4f3c-9b56-549760b8bab3-console-config\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.163812 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.163775    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:52.173934 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.173907    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcbvh\" (UniqueName: \"kubernetes.io/projected/da47f42e-3aa5-4f3c-9b56-549760b8bab3-kube-api-access-kcbvh\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.174024 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.173948    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da47f42e-3aa5-4f3c-9b56-549760b8bab3-service-ca\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.174024 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.173971    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da47f42e-3aa5-4f3c-9b56-549760b8bab3-oauth-serving-cert\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.174024 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.174004    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da47f42e-3aa5-4f3c-9b56-549760b8bab3-console-config\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.174130 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.174052    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da47f42e-3aa5-4f3c-9b56-549760b8bab3-tmp\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.174130 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.174114    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da47f42e-3aa5-4f3c-9b56-549760b8bab3-trusted-ca-bundle\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.174202 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.174139    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da47f42e-3aa5-4f3c-9b56-549760b8bab3-console-serving-cert\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.174202 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.174165    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da47f42e-3aa5-4f3c-9b56-549760b8bab3-console-oauth-config\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.174582 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.174544    5659 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:37:52.174908 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.174884    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da47f42e-3aa5-4f3c-9b56-549760b8bab3-console-config\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.174908 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.174893    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da47f42e-3aa5-4f3c-9b56-549760b8bab3-service-ca\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.175073 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.174920    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da47f42e-3aa5-4f3c-9b56-549760b8bab3-oauth-serving-cert\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.175125 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.175079    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da47f42e-3aa5-4f3c-9b56-549760b8bab3-tmp\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.175425 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.175407    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da47f42e-3aa5-4f3c-9b56-549760b8bab3-trusted-ca-bundle\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.176804 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.176785    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da47f42e-3aa5-4f3c-9b56-549760b8bab3-console-oauth-config\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.176938 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.176921    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da47f42e-3aa5-4f3c-9b56-549760b8bab3-console-serving-cert\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.189333 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.189308    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcbvh\" (UniqueName: \"kubernetes.io/projected/da47f42e-3aa5-4f3c-9b56-549760b8bab3-kube-api-access-kcbvh\") pod \"console-7b6799d47f-t448x\" (UID: \"da47f42e-3aa5-4f3c-9b56-549760b8bab3\") " pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.293872 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.293837    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:52.370303 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.370229    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:37:52.570197 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.570161    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:52.645068 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.644989    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b6799d47f-t448x"]
Nov 03 19:37:52.647950 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:37:52.647919    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda47f42e_3aa5_4f3c_9b56_549760b8bab3.slice/crio-40387bc015e45f1910dae7fc89c872f81b6b6f810da5901ff674b992ea62607a WatchSource:0}: Error finding container 40387bc015e45f1910dae7fc89c872f81b6b6f810da5901ff674b992ea62607a: Status 404 returned error can't find the container with id 40387bc015e45f1910dae7fc89c872f81b6b6f810da5901ff674b992ea62607a
Nov 03 19:37:52.703762 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:52.703735    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:53.112374 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:53.112338    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6799d47f-t448x" event={"ID":"da47f42e-3aa5-4f3c-9b56-549760b8bab3","Type":"ContainerStarted","Data":"69a47a53e3a36fdb8f99ce618c5062bc1d11cc8a55aa11b113cf4a7f8148ae7c"}
Nov 03 19:37:53.112565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:53.112388    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6799d47f-t448x" event={"ID":"da47f42e-3aa5-4f3c-9b56-549760b8bab3","Type":"ContainerStarted","Data":"40387bc015e45f1910dae7fc89c872f81b6b6f810da5901ff674b992ea62607a"}
Nov 03 19:37:53.134819 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:53.134663    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7b6799d47f-t448x" podStartSLOduration=1.134644405 podStartE2EDuration="1.134644405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:53.134634048 +0000 UTC m=+25.289536489" watchObservedRunningTime="2026-11-03 19:37:53.134644405 +0000 UTC m=+25.289546825"
Nov 03 19:37:53.963170 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:53.963111    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6cdcc6bb85-48j6f" podUID="c8236a6c-f408-4996-906d-a73a8cb0069d" containerName="console" containerID="cri-o://dadcf4d7751bc8cea978607d5d6e9357e80705831cfd7918d5faeb18659924c3" gracePeriod=15
Nov 03 19:37:53.965804 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:53.965758    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-67d4ffd4f9-rc92g" podUID="99464aea-b429-4992-b4dd-ad48c7d99633" containerName="console" containerID="cri-o://51b36dea60f2863db390b45cb22a0675844a1ecd8adec21ab000a121eadc7097" gracePeriod=15
Nov 03 19:37:54.118552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.118514    5659 generic.go:356] "Generic (PLEG): container finished" podID="c8236a6c-f408-4996-906d-a73a8cb0069d" containerID="dadcf4d7751bc8cea978607d5d6e9357e80705831cfd7918d5faeb18659924c3" exitCode=0
Nov 03 19:37:54.118729 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.118581    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdcc6bb85-48j6f" event={"ID":"c8236a6c-f408-4996-906d-a73a8cb0069d","Type":"ContainerDied","Data":"dadcf4d7751bc8cea978607d5d6e9357e80705831cfd7918d5faeb18659924c3"}
Nov 03 19:37:54.120208 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.120187    5659 generic.go:356] "Generic (PLEG): container finished" podID="99464aea-b429-4992-b4dd-ad48c7d99633" containerID="51b36dea60f2863db390b45cb22a0675844a1ecd8adec21ab000a121eadc7097" exitCode=0
Nov 03 19:37:54.120328 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.120265    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d4ffd4f9-rc92g" event={"ID":"99464aea-b429-4992-b4dd-ad48c7d99633","Type":"ContainerDied","Data":"51b36dea60f2863db390b45cb22a0675844a1ecd8adec21ab000a121eadc7097"}
Nov 03 19:37:54.254008 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.253985    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:54.256520 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.256503    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:54.296974 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.296942    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99464aea-b429-4992-b4dd-ad48c7d99633-kube-api-access-7knzb\" (UniqueName: \"kubernetes.io/projected/99464aea-b429-4992-b4dd-ad48c7d99633-kube-api-access-7knzb\") pod \"99464aea-b429-4992-b4dd-ad48c7d99633\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") "
Nov 03 19:37:54.297161 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.296991    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99464aea-b429-4992-b4dd-ad48c7d99633-tmp\" (UniqueName: \"kubernetes.io/empty-dir/99464aea-b429-4992-b4dd-ad48c7d99633-tmp\") pod \"99464aea-b429-4992-b4dd-ad48c7d99633\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") "
Nov 03 19:37:54.297161 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297045    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-service-ca\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-service-ca\") pod \"99464aea-b429-4992-b4dd-ad48c7d99633\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") "
Nov 03 19:37:54.297161 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297080    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-trusted-ca-bundle\") pod \"c8236a6c-f408-4996-906d-a73a8cb0069d\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") "
Nov 03 19:37:54.297161 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297111    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8236a6c-f408-4996-906d-a73a8cb0069d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8236a6c-f408-4996-906d-a73a8cb0069d-tmp\") pod \"c8236a6c-f408-4996-906d-a73a8cb0069d\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") "
Nov 03 19:37:54.297161 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297148    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-console-config\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-console-config\") pod \"99464aea-b429-4992-b4dd-ad48c7d99633\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") "
Nov 03 19:37:54.297512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297173    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-oauth-config\") pod \"c8236a6c-f408-4996-906d-a73a8cb0069d\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") "
Nov 03 19:37:54.297512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297215    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-console-config\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-console-config\") pod \"c8236a6c-f408-4996-906d-a73a8cb0069d\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") "
Nov 03 19:37:54.297512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297246    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-serving-cert\") pod \"c8236a6c-f408-4996-906d-a73a8cb0069d\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") "
Nov 03 19:37:54.297512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297278    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-oauth-serving-cert\") pod \"c8236a6c-f408-4996-906d-a73a8cb0069d\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") "
Nov 03 19:37:54.297512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297308    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-trusted-ca-bundle\") pod \"99464aea-b429-4992-b4dd-ad48c7d99633\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") "
Nov 03 19:37:54.297512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297331    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99464aea-b429-4992-b4dd-ad48c7d99633-tmp" pod "99464aea-b429-4992-b4dd-ad48c7d99633" (UID: "99464aea-b429-4992-b4dd-ad48c7d99633"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:54.297512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297337    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-oauth-serving-cert\") pod \"99464aea-b429-4992-b4dd-ad48c7d99633\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") "
Nov 03 19:37:54.297512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297461    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-service-ca" pod "99464aea-b429-4992-b4dd-ad48c7d99633" (UID: "99464aea-b429-4992-b4dd-ad48c7d99633"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:54.297896 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297464    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-serving-cert\") pod \"99464aea-b429-4992-b4dd-ad48c7d99633\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") "
Nov 03 19:37:54.297896 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297654    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-oauth-config\") pod \"99464aea-b429-4992-b4dd-ad48c7d99633\" (UID: \"99464aea-b429-4992-b4dd-ad48c7d99633\") "
Nov 03 19:37:54.297896 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297659    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-oauth-serving-cert" pod "99464aea-b429-4992-b4dd-ad48c7d99633" (UID: "99464aea-b429-4992-b4dd-ad48c7d99633"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:54.297896 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297695    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-service-ca\") pod \"c8236a6c-f408-4996-906d-a73a8cb0069d\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") "
Nov 03 19:37:54.297896 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297734    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8236a6c-f408-4996-906d-a73a8cb0069d-kube-api-access-sj9m7\" (UniqueName: \"kubernetes.io/projected/c8236a6c-f408-4996-906d-a73a8cb0069d-kube-api-access-sj9m7\") pod \"c8236a6c-f408-4996-906d-a73a8cb0069d\" (UID: \"c8236a6c-f408-4996-906d-a73a8cb0069d\") "
Nov 03 19:37:54.298169 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.297998    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8236a6c-f408-4996-906d-a73a8cb0069d-tmp" pod "c8236a6c-f408-4996-906d-a73a8cb0069d" (UID: "c8236a6c-f408-4996-906d-a73a8cb0069d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:54.298169 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.298010    5659 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99464aea-b429-4992-b4dd-ad48c7d99633-tmp\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.298169 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.298068    5659 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.298169 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.298084    5659 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-oauth-serving-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.298169 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.298086    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-oauth-serving-cert" pod "c8236a6c-f408-4996-906d-a73a8cb0069d" (UID: "c8236a6c-f408-4996-906d-a73a8cb0069d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:54.298414 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.298314    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-trusted-ca-bundle" pod "c8236a6c-f408-4996-906d-a73a8cb0069d" (UID: "c8236a6c-f408-4996-906d-a73a8cb0069d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:54.299023 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.298973    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-trusted-ca-bundle" pod "99464aea-b429-4992-b4dd-ad48c7d99633" (UID: "99464aea-b429-4992-b4dd-ad48c7d99633"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:54.299023 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.299008    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-console-config" pod "c8236a6c-f408-4996-906d-a73a8cb0069d" (UID: "c8236a6c-f408-4996-906d-a73a8cb0069d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:54.299236 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.299032    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-service-ca" pod "c8236a6c-f408-4996-906d-a73a8cb0069d" (UID: "c8236a6c-f408-4996-906d-a73a8cb0069d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:54.299297 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.299264    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-console-config" pod "99464aea-b429-4992-b4dd-ad48c7d99633" (UID: "99464aea-b429-4992-b4dd-ad48c7d99633"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:54.301440 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.301414    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8236a6c-f408-4996-906d-a73a8cb0069d-kube-api-access-sj9m7" pod "c8236a6c-f408-4996-906d-a73a8cb0069d" (UID: "c8236a6c-f408-4996-906d-a73a8cb0069d"). InnerVolumeSpecName "kube-api-access-sj9m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:54.301549 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.301436    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99464aea-b429-4992-b4dd-ad48c7d99633-kube-api-access-7knzb" pod "99464aea-b429-4992-b4dd-ad48c7d99633" (UID: "99464aea-b429-4992-b4dd-ad48c7d99633"). InnerVolumeSpecName "kube-api-access-7knzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:54.301549 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.301423    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-serving-cert" pod "c8236a6c-f408-4996-906d-a73a8cb0069d" (UID: "c8236a6c-f408-4996-906d-a73a8cb0069d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:54.302075 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.302048    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-serving-cert" pod "99464aea-b429-4992-b4dd-ad48c7d99633" (UID: "99464aea-b429-4992-b4dd-ad48c7d99633"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:54.302397 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.302374    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-oauth-config" pod "99464aea-b429-4992-b4dd-ad48c7d99633" (UID: "99464aea-b429-4992-b4dd-ad48c7d99633"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:54.302637 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.302621    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-oauth-config" pod "c8236a6c-f408-4996-906d-a73a8cb0069d" (UID: "c8236a6c-f408-4996-906d-a73a8cb0069d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:54.399211 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.399174    5659 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-serving-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.399211 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.399206    5659 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99464aea-b429-4992-b4dd-ad48c7d99633-console-oauth-config\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.399211 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.399215    5659 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.399436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.399225    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj9m7\" (UniqueName: \"kubernetes.io/projected/c8236a6c-f408-4996-906d-a73a8cb0069d-kube-api-access-sj9m7\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.399436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.399235    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7knzb\" (UniqueName: \"kubernetes.io/projected/99464aea-b429-4992-b4dd-ad48c7d99633-kube-api-access-7knzb\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.399436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.399243    5659 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-trusted-ca-bundle\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.399436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.399252    5659 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8236a6c-f408-4996-906d-a73a8cb0069d-tmp\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.399436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.399261    5659 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-console-config\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.399436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.399270    5659 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-oauth-config\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.399436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.399278    5659 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-console-config\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.399436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.399293    5659 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8236a6c-f408-4996-906d-a73a8cb0069d-console-serving-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.399436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.399303    5659 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8236a6c-f408-4996-906d-a73a8cb0069d-oauth-serving-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:54.399436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:54.399311    5659 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99464aea-b429-4992-b4dd-ad48c7d99633-trusted-ca-bundle\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.054455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.054415    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5d4cc48b5f-9g4kc" podUID="51cdd330-78ed-418e-8bdb-1f080d7c2b11" containerName="console" containerID="cri-o://360c65d6bd521802d5b2fa893983456cf9889aece84bf553d31dff0aee76a209" gracePeriod=15
Nov 03 19:37:55.125417 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.125386    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d4ffd4f9-rc92g"
Nov 03 19:37:55.125585 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.125378    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d4ffd4f9-rc92g" event={"ID":"99464aea-b429-4992-b4dd-ad48c7d99633","Type":"ContainerDied","Data":"0c2740c722e154dea4fa7c3ab8387754bcf4654e4dac8361ab8b0c0f5ea0effc"}
Nov 03 19:37:55.125585 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.125558    5659 scope.go:122] "RemoveContainer" containerID="51b36dea60f2863db390b45cb22a0675844a1ecd8adec21ab000a121eadc7097"
Nov 03 19:37:55.126878 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.126803    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdcc6bb85-48j6f" event={"ID":"c8236a6c-f408-4996-906d-a73a8cb0069d","Type":"ContainerDied","Data":"1eb57135de286a127233d228fa622464972983143f5729150279ebd5289647dc"}
Nov 03 19:37:55.126878 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.126833    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdcc6bb85-48j6f"
Nov 03 19:37:55.152188 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.152159    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cdcc6bb85-48j6f"]
Nov 03 19:37:55.157848 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.157818    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cdcc6bb85-48j6f"]
Nov 03 19:37:55.164891 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.164867    5659 scope.go:122] "RemoveContainer" containerID="dadcf4d7751bc8cea978607d5d6e9357e80705831cfd7918d5faeb18659924c3"
Nov 03 19:37:55.170712 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.170690    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d4ffd4f9-rc92g"]
Nov 03 19:37:55.174460 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.174438    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67d4ffd4f9-rc92g"]
Nov 03 19:37:55.357009 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.356978    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:55.406424 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.406396    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cdd330-78ed-418e-8bdb-1f080d7c2b11-kube-api-access-fs4rw\" (UniqueName: \"kubernetes.io/projected/51cdd330-78ed-418e-8bdb-1f080d7c2b11-kube-api-access-fs4rw\") pod \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") "
Nov 03 19:37:55.406632 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.406449    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-serving-cert\") pod \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") "
Nov 03 19:37:55.406632 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.406470    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-service-ca\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-service-ca\") pod \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") "
Nov 03 19:37:55.406632 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.406512    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-oauth-config\") pod \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") "
Nov 03 19:37:55.406802 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.406682    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-oauth-serving-cert\") pod \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") "
Nov 03 19:37:55.406802 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.406773    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-config\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-config\") pod \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") "
Nov 03 19:37:55.406906 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.406808    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cdd330-78ed-418e-8bdb-1f080d7c2b11-tmp\" (UniqueName: \"kubernetes.io/empty-dir/51cdd330-78ed-418e-8bdb-1f080d7c2b11-tmp\") pod \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\" (UID: \"51cdd330-78ed-418e-8bdb-1f080d7c2b11\") "
Nov 03 19:37:55.406973 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.406953    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-service-ca" pod "51cdd330-78ed-418e-8bdb-1f080d7c2b11" (UID: "51cdd330-78ed-418e-8bdb-1f080d7c2b11"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:55.407060 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.407035    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-oauth-serving-cert" pod "51cdd330-78ed-418e-8bdb-1f080d7c2b11" (UID: "51cdd330-78ed-418e-8bdb-1f080d7c2b11"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:55.407216 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.407167    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cdd330-78ed-418e-8bdb-1f080d7c2b11-tmp" pod "51cdd330-78ed-418e-8bdb-1f080d7c2b11" (UID: "51cdd330-78ed-418e-8bdb-1f080d7c2b11"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:55.407216 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.407180    5659 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.407216 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.407192    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-config" pod "51cdd330-78ed-418e-8bdb-1f080d7c2b11" (UID: "51cdd330-78ed-418e-8bdb-1f080d7c2b11"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:55.407216 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.407205    5659 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-oauth-serving-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.408773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.408751    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-oauth-config" pod "51cdd330-78ed-418e-8bdb-1f080d7c2b11" (UID: "51cdd330-78ed-418e-8bdb-1f080d7c2b11"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:55.408854 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.408806    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cdd330-78ed-418e-8bdb-1f080d7c2b11-kube-api-access-fs4rw" pod "51cdd330-78ed-418e-8bdb-1f080d7c2b11" (UID: "51cdd330-78ed-418e-8bdb-1f080d7c2b11"). InnerVolumeSpecName "kube-api-access-fs4rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:55.408854 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.408817    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-serving-cert" pod "51cdd330-78ed-418e-8bdb-1f080d7c2b11" (UID: "51cdd330-78ed-418e-8bdb-1f080d7c2b11"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:55.508051 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.508008    5659 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-config\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.508051 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.508040    5659 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51cdd330-78ed-418e-8bdb-1f080d7c2b11-tmp\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.508051 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.508050    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs4rw\" (UniqueName: \"kubernetes.io/projected/51cdd330-78ed-418e-8bdb-1f080d7c2b11-kube-api-access-fs4rw\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.508051 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.508061    5659 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-serving-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.508314 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.508070    5659 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51cdd330-78ed-418e-8bdb-1f080d7c2b11-console-oauth-config\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.708024 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.707993    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:55.845259 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.845223    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:55.968759 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.968681    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99464aea-b429-4992-b4dd-ad48c7d99633" path="/var/lib/kubelet/pods/99464aea-b429-4992-b4dd-ad48c7d99633/volumes"
Nov 03 19:37:55.969094 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:55.969080    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8236a6c-f408-4996-906d-a73a8cb0069d" path="/var/lib/kubelet/pods/c8236a6c-f408-4996-906d-a73a8cb0069d/volumes"
Nov 03 19:37:56.132183 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:56.132148    5659 generic.go:356] "Generic (PLEG): container finished" podID="51cdd330-78ed-418e-8bdb-1f080d7c2b11" containerID="360c65d6bd521802d5b2fa893983456cf9889aece84bf553d31dff0aee76a209" exitCode=0
Nov 03 19:37:56.132662 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:56.132219    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4cc48b5f-9g4kc"
Nov 03 19:37:56.132662 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:56.132235    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4cc48b5f-9g4kc" event={"ID":"51cdd330-78ed-418e-8bdb-1f080d7c2b11","Type":"ContainerDied","Data":"360c65d6bd521802d5b2fa893983456cf9889aece84bf553d31dff0aee76a209"}
Nov 03 19:37:56.132662 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:56.132286    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4cc48b5f-9g4kc" event={"ID":"51cdd330-78ed-418e-8bdb-1f080d7c2b11","Type":"ContainerDied","Data":"8e9861ceb2125342bfad482387a49b5d772d504601db52983f25766fb5420c3a"}
Nov 03 19:37:56.132662 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:56.132315    5659 scope.go:122] "RemoveContainer" containerID="360c65d6bd521802d5b2fa893983456cf9889aece84bf553d31dff0aee76a209"
Nov 03 19:37:56.139926 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:56.139907    5659 scope.go:122] "RemoveContainer" containerID="360c65d6bd521802d5b2fa893983456cf9889aece84bf553d31dff0aee76a209"
Nov 03 19:37:56.140205 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:56.140185    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"360c65d6bd521802d5b2fa893983456cf9889aece84bf553d31dff0aee76a209\": container with ID starting with 360c65d6bd521802d5b2fa893983456cf9889aece84bf553d31dff0aee76a209 not found: ID does not exist" containerID="360c65d6bd521802d5b2fa893983456cf9889aece84bf553d31dff0aee76a209"
Nov 03 19:37:56.140273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:56.140219    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360c65d6bd521802d5b2fa893983456cf9889aece84bf553d31dff0aee76a209"} err="failed to get container status \"360c65d6bd521802d5b2fa893983456cf9889aece84bf553d31dff0aee76a209\": rpc error: code = NotFound desc = could not find container \"360c65d6bd521802d5b2fa893983456cf9889aece84bf553d31dff0aee76a209\": container with ID starting with 360c65d6bd521802d5b2fa893983456cf9889aece84bf553d31dff0aee76a209 not found: ID does not exist"
Nov 03 19:37:56.153478 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:56.153454    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d4cc48b5f-9g4kc"]
Nov 03 19:37:56.160688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:56.160664    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d4cc48b5f-9g4kc"]
Nov 03 19:37:56.452563 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:56.452492    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vcw9t"
Nov 03 19:37:56.705547 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:56.705468    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:56.907422 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:56.907387    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:57.231322 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:57.231295    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6q6gg"
Nov 03 19:37:57.966939 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:57.966903    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51cdd330-78ed-418e-8bdb-1f080d7c2b11" path="/var/lib/kubelet/pods/51cdd330-78ed-418e-8bdb-1f080d7c2b11/volumes"
Nov 03 19:37:57.967227 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:57.967210    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:37:57.967545 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:57.967523    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="prometheus" containerID="cri-o://bbdd7e2ec527a85de1cad1e4981a42589b867a0ccc3d730013a818cca3d2adb4" gracePeriod=600
Nov 03 19:37:57.967623 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:57.967596    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="kube-rbac-proxy" containerID="cri-o://c7400fea3ba057666ea68664c8c083a9f1531956c7284e97fb18a0ee40503e5b" gracePeriod=600
Nov 03 19:37:57.967684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:57.967616    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="thanos-sidecar" containerID="cri-o://d8955bb3913556a7dcf78ded1a893ac224703af86c2c9254f99ac6d552bc4b08" gracePeriod=600
Nov 03 19:37:57.967684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:57.967618    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bf51b150c27764b3bdee088353bc5a57232cd08bd66985b19dce786935f1c1ea" gracePeriod=600
Nov 03 19:37:57.967785 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:57.967656    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="config-reloader" containerID="cri-o://0189b7f6d3ed356b8453d577364ebc70a887771aafae13ffed30c7e494a1d878" gracePeriod=600
Nov 03 19:37:57.967785 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:57.967670    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="kube-rbac-proxy-web" containerID="cri-o://ec9eb9115176013e15a5c1fb9a8470723d3e4c3f26c228e2984e46487473debd" gracePeriod=600
Nov 03 19:37:58.143811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:58.143781    5659 generic.go:356] "Generic (PLEG): container finished" podID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerID="bf51b150c27764b3bdee088353bc5a57232cd08bd66985b19dce786935f1c1ea" exitCode=0
Nov 03 19:37:58.143811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:58.143807    5659 generic.go:356] "Generic (PLEG): container finished" podID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerID="c7400fea3ba057666ea68664c8c083a9f1531956c7284e97fb18a0ee40503e5b" exitCode=0
Nov 03 19:37:58.143811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:58.143814    5659 generic.go:356] "Generic (PLEG): container finished" podID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerID="d8955bb3913556a7dcf78ded1a893ac224703af86c2c9254f99ac6d552bc4b08" exitCode=0
Nov 03 19:37:58.143811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:58.143821    5659 generic.go:356] "Generic (PLEG): container finished" podID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerID="0189b7f6d3ed356b8453d577364ebc70a887771aafae13ffed30c7e494a1d878" exitCode=0
Nov 03 19:37:58.144048 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:58.143832    5659 generic.go:356] "Generic (PLEG): container finished" podID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerID="bbdd7e2ec527a85de1cad1e4981a42589b867a0ccc3d730013a818cca3d2adb4" exitCode=0
Nov 03 19:37:58.144048 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:58.143849    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerDied","Data":"bf51b150c27764b3bdee088353bc5a57232cd08bd66985b19dce786935f1c1ea"}
Nov 03 19:37:58.144048 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:58.143919    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerDied","Data":"c7400fea3ba057666ea68664c8c083a9f1531956c7284e97fb18a0ee40503e5b"}
Nov 03 19:37:58.144048 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:58.143946    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerDied","Data":"d8955bb3913556a7dcf78ded1a893ac224703af86c2c9254f99ac6d552bc4b08"}
Nov 03 19:37:58.144048 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:58.143971    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerDied","Data":"0189b7f6d3ed356b8453d577364ebc70a887771aafae13ffed30c7e494a1d878"}
Nov 03 19:37:58.144048 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:58.143995    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerDied","Data":"bbdd7e2ec527a85de1cad1e4981a42589b867a0ccc3d730013a818cca3d2adb4"}
Nov 03 19:37:58.504915 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:58.504872    5659 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="bbdd7e2ec527a85de1cad1e4981a42589b867a0ccc3d730013a818cca3d2adb4" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 03 19:37:58.505267 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:58.504928    5659 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="bbdd7e2ec527a85de1cad1e4981a42589b867a0ccc3d730013a818cca3d2adb4" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 03 19:37:58.505267 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:37:58.504944    5659 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="bbdd7e2ec527a85de1cad1e4981a42589b867a0ccc3d730013a818cca3d2adb4" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 03 19:37:58.505267 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:58.504957    5659 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Nov 03 19:37:58.783044 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:58.782960    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:59.153697 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.152315    5659 generic.go:356] "Generic (PLEG): container finished" podID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerID="ec9eb9115176013e15a5c1fb9a8470723d3e4c3f26c228e2984e46487473debd" exitCode=0
Nov 03 19:37:59.153697 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.152377    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerDied","Data":"ec9eb9115176013e15a5c1fb9a8470723d3e4c3f26c228e2984e46487473debd"}
Nov 03 19:37:59.245812 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.245779    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:37:59.340773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.340686    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-thanos-prometheus-http-client-file\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.340773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.340736    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-2\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.340773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.340759    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config-out\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.341047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.340788    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-metrics-client-ca\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.341047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.340817    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-0\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.341047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.340839    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-kube-rbac-proxy\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.341450 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.341254    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-metrics-client-ca" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:59.341593 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.341564    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-2" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:59.341668 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.341624    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-serving-certs-ca-bundle\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.341729 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.341663    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-serving-certs-ca-bundle" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:59.341729 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.341690    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-tls-assets\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-tls-assets\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.341831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.341725    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-kube-api-access-d6k67\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-kube-api-access-d6k67\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.341831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.341754    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-tls\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.341831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.341795    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-grpc-tls\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.341831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.341825    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-1\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.342044 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.341854    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.342044 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.341887    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-db\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.342044 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.341937    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.342044 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.341971    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-metrics-client-certs\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.342044 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.342015    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-trusted-ca-bundle\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.342295 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.342046    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.342295 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.342077    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-web-config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-web-config\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.342295 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.342114    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-kubelet-serving-ca-bundle\") pod \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\" (UID: \"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec\") "
Nov 03 19:37:59.342447 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.342308    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-1" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:59.342447 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.342384    5659 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.342447 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.342405    5659 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-metrics-client-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.342447 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.342421    5659 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.342447 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.342435    5659 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.342836 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.342671    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-kubelet-serving-ca-bundle" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:59.342898 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.342844    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-0" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:59.343918 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.343654    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-trusted-ca-bundle" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:59.343918 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.343758    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config-out" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:59.344131 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.343970    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-db" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:59.344503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.344428    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-thanos-prometheus-http-client-file" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:59.345691 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.345660    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-kube-api-access-d6k67" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "kube-api-access-d6k67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:59.346158 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.346120    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-grpc-tls" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:59.346158 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.346149    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-tls-assets" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:59.346476 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.346447    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-tls" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:59.346762 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.346737    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-kube-rbac-proxy" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:59.346870 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.346836    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-metrics-client-certs" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:59.346929 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.346903    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:59.347034 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.347013    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-kube-rbac-proxy-web" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:59.347555 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.347532    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-thanos-sidecar-tls" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:59.356441 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.356416    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-web-config" pod "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" (UID: "6007b5bf-fe4e-47e8-b6d6-57f4b03768ec"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:59.442798 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442755    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6k67\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-kube-api-access-d6k67\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.442798 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442791    5659 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.442798 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442801    5659 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-grpc-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.443026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442810    5659 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.443026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442821    5659 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-db\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.443026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442830    5659 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.443026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442839    5659 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-metrics-client-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.443026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442848    5659 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.443026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442857    5659 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.443026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442890    5659 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-web-config\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.443026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442901    5659 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.443026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442910    5659 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.443026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442919    5659 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-config-out\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.443026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442927    5659 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.443026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442935    5659 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-secret-kube-rbac-proxy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.443026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:37:59.442943    5659 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec-tls-assets\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:00.158590 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.158553    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6007b5bf-fe4e-47e8-b6d6-57f4b03768ec","Type":"ContainerDied","Data":"41dfb349eab98c7c8e5cfe4b15803d9fb5393310c0f09b38bcf56d1d9f43273b"}
Nov 03 19:38:00.158977 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.158636    5659 scope.go:122] "RemoveContainer" containerID="bf51b150c27764b3bdee088353bc5a57232cd08bd66985b19dce786935f1c1ea"
Nov 03 19:38:00.158977 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.158643    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.165860 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.165843    5659 scope.go:122] "RemoveContainer" containerID="c7400fea3ba057666ea68664c8c083a9f1531956c7284e97fb18a0ee40503e5b"
Nov 03 19:38:00.172303 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.172282    5659 scope.go:122] "RemoveContainer" containerID="ec9eb9115176013e15a5c1fb9a8470723d3e4c3f26c228e2984e46487473debd"
Nov 03 19:38:00.178514 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.178477    5659 scope.go:122] "RemoveContainer" containerID="d8955bb3913556a7dcf78ded1a893ac224703af86c2c9254f99ac6d552bc4b08"
Nov 03 19:38:00.180169 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.180150    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:38:00.185707 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.185691    5659 scope.go:122] "RemoveContainer" containerID="0189b7f6d3ed356b8453d577364ebc70a887771aafae13ffed30c7e494a1d878"
Nov 03 19:38:00.185799 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.185769    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:38:00.191878 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.191854    5659 scope.go:122] "RemoveContainer" containerID="bbdd7e2ec527a85de1cad1e4981a42589b867a0ccc3d730013a818cca3d2adb4"
Nov 03 19:38:00.198544 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.198528    5659 scope.go:122] "RemoveContainer" containerID="208cb05cb678e62705b2663440c26a5ccc690ce877d907c6a205f699b8c76ae1"
Nov 03 19:38:00.217247 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217223    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:38:00.217533 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217518    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="kube-rbac-proxy-web"
Nov 03 19:38:00.217598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217533    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="kube-rbac-proxy-web"
Nov 03 19:38:00.217598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217548    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="kube-rbac-proxy-thanos"
Nov 03 19:38:00.217598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217555    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="kube-rbac-proxy-thanos"
Nov 03 19:38:00.217598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217569    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="thanos-sidecar"
Nov 03 19:38:00.217598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217574    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="thanos-sidecar"
Nov 03 19:38:00.217598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217583    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="init-config-reloader"
Nov 03 19:38:00.217598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217587    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="init-config-reloader"
Nov 03 19:38:00.217598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217596    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="kube-rbac-proxy"
Nov 03 19:38:00.217598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217601    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="kube-rbac-proxy"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217609    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c8236a6c-f408-4996-906d-a73a8cb0069d" containerName="console"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217613    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8236a6c-f408-4996-906d-a73a8cb0069d" containerName="console"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217619    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="config-reloader"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217623    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="config-reloader"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217629    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="prometheus"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217633    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="prometheus"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217639    5659 cpu_manager.go:414] "containerMap: removing container" podUID="99464aea-b429-4992-b4dd-ad48c7d99633" containerName="console"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217643    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99464aea-b429-4992-b4dd-ad48c7d99633" containerName="console"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217648    5659 cpu_manager.go:414] "containerMap: removing container" podUID="51cdd330-78ed-418e-8bdb-1f080d7c2b11" containerName="console"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217652    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51cdd330-78ed-418e-8bdb-1f080d7c2b11" containerName="console"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217696    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="kube-rbac-proxy-web"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217704    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="kube-rbac-proxy-thanos"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217710    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="thanos-sidecar"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217715    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="init-config-reloader"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217721    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="kube-rbac-proxy"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217728    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8236a6c-f408-4996-906d-a73a8cb0069d" containerName="console"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217733    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="config-reloader"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217738    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" containerName="prometheus"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217744    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="99464aea-b429-4992-b4dd-ad48c7d99633" containerName="console"
Nov 03 19:38:00.217879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.217748    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="51cdd330-78ed-418e-8bdb-1f080d7c2b11" containerName="console"
Nov 03 19:38:00.244706 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.244650    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:38:00.244806 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.244790    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.253670 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.253650    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 19:38:00.253775 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.253686    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 19:38:00.253948 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.253848    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 19:38:00.253948 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.253885    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 19:38:00.253948 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.253893    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 19:38:00.253948 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.253904    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 19:38:00.253948 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.253906    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 19:38:00.253948 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.253922    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 19:38:00.253948 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.253895    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 19:38:00.253948 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.253940    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 19:38:00.254894 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.254870    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zljmq\""
Nov 03 19:38:00.254966 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.254913    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 19:38:00.254966 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.254955    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 19:38:00.255026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.254995    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1kr3opstr93m9\""
Nov 03 19:38:00.260431 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.260415    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 19:38:00.260570 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.260555    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 19:38:00.350140 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350106    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350140 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350140    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350390 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350157    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350390 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350175    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-web-config\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350390 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350215    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350390 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350297    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da5c9072-3ba7-49da-8c2f-f81374cd3b81-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350390 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350376    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350418    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da5c9072-3ba7-49da-8c2f-f81374cd3b81-config-out\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350447    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350506    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-config\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350523    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350537    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350556    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350646    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350962 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350687    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350962 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350720    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350962 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350804    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjxz2\" (UniqueName: \"kubernetes.io/projected/da5c9072-3ba7-49da-8c2f-f81374cd3b81-kube-api-access-zjxz2\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350962 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350833    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350962 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350854    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.350962 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.350869    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.451290 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.451257    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.451290 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.451301    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.451500 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.451318    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.451500 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.451384    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.451500 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.451428    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.451500 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.451460    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.451655 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.451500    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-web-config\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.451655 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.451522    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.451655 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.451548    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da5c9072-3ba7-49da-8c2f-f81374cd3b81-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.452879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.452526    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.452879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.452573    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.452879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.452595    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.453564 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.453541    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.453832 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.453812    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.454054 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.454036    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da5c9072-3ba7-49da-8c2f-f81374cd3b81-config-out\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.454186 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.454170    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.454298 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.454285    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-config\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.454404 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.454390    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.454524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.454509    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.454648 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.454634    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.454777 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.454762    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.454934 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.454897    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.455051 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.455038    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.455229 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.455205    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-web-config\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.455348 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.455141    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjxz2\" (UniqueName: \"kubernetes.io/projected/da5c9072-3ba7-49da-8c2f-f81374cd3b81-kube-api-access-zjxz2\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.460203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.455664    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.460203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.454813    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da5c9072-3ba7-49da-8c2f-f81374cd3b81-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.460203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.454931    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.460203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.455861    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.460203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.456642    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.460203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.457505    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da5c9072-3ba7-49da-8c2f-f81374cd3b81-config-out\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.460203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.454763    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.460203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.457757    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-config\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.460203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.457814    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.460203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.458127    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.460203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.458473    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.460203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.459130    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.460836 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.460556    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da5c9072-3ba7-49da-8c2f-f81374cd3b81-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.461884 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.461853    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da5c9072-3ba7-49da-8c2f-f81374cd3b81-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.470932 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.470909    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjxz2\" (UniqueName: \"kubernetes.io/projected/da5c9072-3ba7-49da-8c2f-f81374cd3b81-kube-api-access-zjxz2\") pod \"prometheus-k8s-0\" (UID: \"da5c9072-3ba7-49da-8c2f-f81374cd3b81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.611880 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.611798    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:00.932997 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:00.932969    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 19:38:00.934836 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:38:00.934795    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5c9072_3ba7_49da_8c2f_f81374cd3b81.slice/crio-407e50acd80806e9659f662a1678e48523b5269537a4b4c000e53158ffe7fd7a WatchSource:0}: Error finding container 407e50acd80806e9659f662a1678e48523b5269537a4b4c000e53158ffe7fd7a: Status 404 returned error can't find the container with id 407e50acd80806e9659f662a1678e48523b5269537a4b4c000e53158ffe7fd7a
Nov 03 19:38:01.163676 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:01.163643    5659 generic.go:356] "Generic (PLEG): container finished" podID="da5c9072-3ba7-49da-8c2f-f81374cd3b81" containerID="cc9d61784cb4a956a18073c998d16fb7563f06c2eaf3e4b58e56b11ed3334351" exitCode=0
Nov 03 19:38:01.164129 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:01.163733    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5c9072-3ba7-49da-8c2f-f81374cd3b81","Type":"ContainerDied","Data":"cc9d61784cb4a956a18073c998d16fb7563f06c2eaf3e4b58e56b11ed3334351"}
Nov 03 19:38:01.164129 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:01.163813    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5c9072-3ba7-49da-8c2f-f81374cd3b81","Type":"ContainerStarted","Data":"407e50acd80806e9659f662a1678e48523b5269537a4b4c000e53158ffe7fd7a"}
Nov 03 19:38:01.966817 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:01.966782    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6007b5bf-fe4e-47e8-b6d6-57f4b03768ec" path="/var/lib/kubelet/pods/6007b5bf-fe4e-47e8-b6d6-57f4b03768ec/volumes"
Nov 03 19:38:02.171795 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:02.171712    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5c9072-3ba7-49da-8c2f-f81374cd3b81","Type":"ContainerStarted","Data":"65910933631f6175b12b3202882ad50e88d5947d7ba92fca3a14d7b955560b2e"}
Nov 03 19:38:02.171795 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:02.171793    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5c9072-3ba7-49da-8c2f-f81374cd3b81","Type":"ContainerStarted","Data":"960611d4f93b6aefd991e7cf12d4c792ab63c742509a30cd7a747ef74d117ed3"}
Nov 03 19:38:02.172258 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:02.171833    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5c9072-3ba7-49da-8c2f-f81374cd3b81","Type":"ContainerStarted","Data":"00fefa6a4808e8f7c6f630f7d648a73710b3752eed6507aba7c8cc8f3e3d284b"}
Nov 03 19:38:02.172258 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:02.171870    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5c9072-3ba7-49da-8c2f-f81374cd3b81","Type":"ContainerStarted","Data":"4e7f8a3c0283278c989bec10dcc3afc8772f84d4f2276e0ba7821153a108d58b"}
Nov 03 19:38:02.172258 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:02.171912    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5c9072-3ba7-49da-8c2f-f81374cd3b81","Type":"ContainerStarted","Data":"a1fea62195db137438f16b9dc3522aec71f7b6bd9e7c482356bc898490ca466e"}
Nov 03 19:38:02.172258 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:02.171946    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5c9072-3ba7-49da-8c2f-f81374cd3b81","Type":"ContainerStarted","Data":"4e72e3c5176129521704c4ac906321e43edb6c4117fdab6795e6cdc1dbce55fb"}
Nov 03 19:38:02.214615 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:02.214559    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.197515577 podStartE2EDuration="2.214546066s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.030489ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:02.213054108 +0000 UTC m=+34.367956527" watchObservedRunningTime="2026-11-03 19:38:02.214546066 +0000 UTC m=+34.369448482"
Nov 03 19:38:02.370352 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:02.370313    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:38:02.370558 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:02.370374    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:38:02.375263 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:02.375241    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:38:03.179270 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:03.179239    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b6799d47f-t448x"
Nov 03 19:38:03.226846 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:03.226812    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d6b976d69-rjgm5"]
Nov 03 19:38:05.611963 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:05.611923    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:07.078333 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:07.078299    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:07.209138 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:07.209105    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:10.727614 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:10.727577    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:10.857564 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:10.857532    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:15.217227 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:38:15.217186    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:38:28.246511 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.246429    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7d6b976d69-rjgm5" podUID="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" containerName="console" containerID="cri-o://809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b" gracePeriod=15
Nov 03 19:38:28.284016 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:38:28.283980    5659 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce3bacc_fa93_4880_983e_ac1b2ae7e488.slice/crio-conmon-809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:38:28.487819 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.487797    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:38:28.596298 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.596217    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-service-ca\") pod \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") "
Nov 03 19:38:28.596298 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.596258    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-serving-cert\") pod \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") "
Nov 03 19:38:28.596298 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.596294    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-oauth-config\") pod \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") "
Nov 03 19:38:28.596603 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.596319    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-config\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-config\") pod \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") "
Nov 03 19:38:28.596603 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.596343    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-trusted-ca-bundle\") pod \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") "
Nov 03 19:38:28.596603 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.596366    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-kube-api-access-xrmfq\" (UniqueName: \"kubernetes.io/projected/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-kube-api-access-xrmfq\") pod \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") "
Nov 03 19:38:28.596603 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.596396    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-oauth-serving-cert\") pod \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") "
Nov 03 19:38:28.596603 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.596435    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-tmp\") pod \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\" (UID: \"1ce3bacc-fa93-4880-983e-ac1b2ae7e488\") "
Nov 03 19:38:28.596859 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.596684    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-service-ca" pod "1ce3bacc-fa93-4880-983e-ac1b2ae7e488" (UID: "1ce3bacc-fa93-4880-983e-ac1b2ae7e488"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:38:28.596859 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.596791    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-tmp" pod "1ce3bacc-fa93-4880-983e-ac1b2ae7e488" (UID: "1ce3bacc-fa93-4880-983e-ac1b2ae7e488"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:38:28.596967 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.596863    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-oauth-serving-cert" pod "1ce3bacc-fa93-4880-983e-ac1b2ae7e488" (UID: "1ce3bacc-fa93-4880-983e-ac1b2ae7e488"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:38:28.597023 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.596990    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-trusted-ca-bundle" pod "1ce3bacc-fa93-4880-983e-ac1b2ae7e488" (UID: "1ce3bacc-fa93-4880-983e-ac1b2ae7e488"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:38:28.597075 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.597053    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-config" pod "1ce3bacc-fa93-4880-983e-ac1b2ae7e488" (UID: "1ce3bacc-fa93-4880-983e-ac1b2ae7e488"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:38:28.598559 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.598530    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-serving-cert" pod "1ce3bacc-fa93-4880-983e-ac1b2ae7e488" (UID: "1ce3bacc-fa93-4880-983e-ac1b2ae7e488"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:38:28.598687 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.598660    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-oauth-config" pod "1ce3bacc-fa93-4880-983e-ac1b2ae7e488" (UID: "1ce3bacc-fa93-4880-983e-ac1b2ae7e488"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:38:28.598729 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.598670    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-kube-api-access-xrmfq" pod "1ce3bacc-fa93-4880-983e-ac1b2ae7e488" (UID: "1ce3bacc-fa93-4880-983e-ac1b2ae7e488"). InnerVolumeSpecName "kube-api-access-xrmfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:38:28.697641 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.697608    5659 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:28.697641 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.697638    5659 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-serving-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:28.697823 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.697652    5659 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-oauth-config\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:28.697823 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.697664    5659 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-console-config\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:28.697823 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.697675    5659 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-trusted-ca-bundle\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:28.697823 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.697688    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrmfq\" (UniqueName: \"kubernetes.io/projected/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-kube-api-access-xrmfq\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:28.697823 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.697702    5659 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-oauth-serving-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:28.697823 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:28.697714    5659 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ce3bacc-fa93-4880-983e-ac1b2ae7e488-tmp\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:29.263411 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:29.263372    5659 generic.go:356] "Generic (PLEG): container finished" podID="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" containerID="809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b" exitCode=0
Nov 03 19:38:29.263872 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:29.263415    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6b976d69-rjgm5"
Nov 03 19:38:29.263872 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:29.263434    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6b976d69-rjgm5" event={"ID":"1ce3bacc-fa93-4880-983e-ac1b2ae7e488","Type":"ContainerDied","Data":"809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b"}
Nov 03 19:38:29.263872 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:29.263474    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6b976d69-rjgm5" event={"ID":"1ce3bacc-fa93-4880-983e-ac1b2ae7e488","Type":"ContainerDied","Data":"a85fa15687989263b0d138222aa85fea42b528f5b63b41cf9c82060381afeeb8"}
Nov 03 19:38:29.263872 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:29.263514    5659 scope.go:122] "RemoveContainer" containerID="809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b"
Nov 03 19:38:29.271494 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:29.271463    5659 scope.go:122] "RemoveContainer" containerID="809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b"
Nov 03 19:38:29.271785 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:38:29.271765    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b\": container with ID starting with 809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b not found: ID does not exist" containerID="809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b"
Nov 03 19:38:29.271831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:29.271795    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b"} err="failed to get container status \"809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b\": rpc error: code = NotFound desc = could not find container \"809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b\": container with ID starting with 809a1e7981c2ab505e68507a4bb8fe12b36c436b00802a19e3053e786179034b not found: ID does not exist"
Nov 03 19:38:29.283794 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:29.283772    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d6b976d69-rjgm5"]
Nov 03 19:38:29.287472 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:29.287442    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d6b976d69-rjgm5"]
Nov 03 19:38:29.965308 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:29.965274    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" path="/var/lib/kubelet/pods/1ce3bacc-fa93-4880-983e-ac1b2ae7e488/volumes"
Nov 03 19:38:34.330794 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:34.330758    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:44.491525 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:44.491465    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:44.506367 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:44.506343    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:38:45.214593 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:38:45.214555    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:38:45.341451 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:38:45.341425    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 19:39:15.214811 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:39:15.214732    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:39:22.040134 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:39:22.040099    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:22.172687 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:39:22.172653    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:34.390946 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:39:34.390906    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:45.215385 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:39:45.215347    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:39:55.923938 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:39:55.923899    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:56.066087 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:39:56.066045    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:15.214835 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:40:15.214802    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:40:33.939855 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:40:33.939819    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:34.068334 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:40:34.068282    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:34.439158 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:40:34.439124    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:36.880527 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:40:36.880477    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:37.015813 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:40:37.015779    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:45.215627 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:40:45.215554    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:41:15.216187 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:41:15.216146    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:41:15.887291 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:15.887253    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:17.309761 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:17.309718    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:17.467697 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:17.467656    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:17.598599 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:17.598510    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:18.709298 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:18.709259    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:19.672233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.672199    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"]
Nov 03 19:41:19.672465 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.672453    5659 cpu_manager.go:414] "containerMap: removing container" podUID="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" containerName="console"
Nov 03 19:41:19.672530 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.672466    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" containerName="console"
Nov 03 19:41:19.672566 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.672550    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce3bacc-fa93-4880-983e-ac1b2ae7e488" containerName="console"
Nov 03 19:41:19.675521 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.675502    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"
Nov 03 19:41:19.680312 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.680277    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 19:41:19.680448 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.680316    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 19:41:19.680448 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.680349    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-z4kf9\""
Nov 03 19:41:19.684941 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.684919    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"]
Nov 03 19:41:19.732298 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.732264    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkw8s\" (UniqueName: \"kubernetes.io/projected/05644f4e-db2b-48e8-a640-86c4e03fc043-kube-api-access-rkw8s\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh\" (UID: \"05644f4e-db2b-48e8-a640-86c4e03fc043\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"
Nov 03 19:41:19.732712 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.732312    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh\" (UID: \"05644f4e-db2b-48e8-a640-86c4e03fc043\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"
Nov 03 19:41:19.732712 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.732418    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh\" (UID: \"05644f4e-db2b-48e8-a640-86c4e03fc043\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"
Nov 03 19:41:19.833044 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.833012    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh\" (UID: \"05644f4e-db2b-48e8-a640-86c4e03fc043\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"
Nov 03 19:41:19.833184 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.833054    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkw8s\" (UniqueName: \"kubernetes.io/projected/05644f4e-db2b-48e8-a640-86c4e03fc043-kube-api-access-rkw8s\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh\" (UID: \"05644f4e-db2b-48e8-a640-86c4e03fc043\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"
Nov 03 19:41:19.833184 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.833104    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh\" (UID: \"05644f4e-db2b-48e8-a640-86c4e03fc043\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"
Nov 03 19:41:19.833476 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.833456    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh\" (UID: \"05644f4e-db2b-48e8-a640-86c4e03fc043\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"
Nov 03 19:41:19.833539 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.833468    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh\" (UID: \"05644f4e-db2b-48e8-a640-86c4e03fc043\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"
Nov 03 19:41:19.845753 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.845723    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkw8s\" (UniqueName: \"kubernetes.io/projected/05644f4e-db2b-48e8-a640-86c4e03fc043-kube-api-access-rkw8s\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh\" (UID: \"05644f4e-db2b-48e8-a640-86c4e03fc043\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"
Nov 03 19:41:19.991212 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:19.991182    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"
Nov 03 19:41:20.261976 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:20.261904    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"]
Nov 03 19:41:20.265185 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:41:20.265157    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05644f4e_db2b_48e8_a640_86c4e03fc043.slice/crio-cc35217a5ebfdc5256c6e37d1529c2f784c17ae99f80d44231186a6e5f2b9242 WatchSource:0}: Error finding container cc35217a5ebfdc5256c6e37d1529c2f784c17ae99f80d44231186a6e5f2b9242: Status 404 returned error can't find the container with id cc35217a5ebfdc5256c6e37d1529c2f784c17ae99f80d44231186a6e5f2b9242
Nov 03 19:41:20.267212 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:20.267194    5659 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:41:20.822795 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:20.822761    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh" event={"ID":"05644f4e-db2b-48e8-a640-86c4e03fc043","Type":"ContainerStarted","Data":"cc35217a5ebfdc5256c6e37d1529c2f784c17ae99f80d44231186a6e5f2b9242"}
Nov 03 19:41:22.449933 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.449901    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:22.573806 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.573778    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:22.758048 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.758015    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"]
Nov 03 19:41:22.761605 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.761587    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"
Nov 03 19:41:22.766633 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.766616    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-klkmd\""
Nov 03 19:41:22.771815 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.771796    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"]
Nov 03 19:41:22.846767 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.846735    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"]
Nov 03 19:41:22.850056 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.850040    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"
Nov 03 19:41:22.857110 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.857079    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn\" (UID: \"34ce7d00-e245-4eab-b9c9-6c9580000443\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"
Nov 03 19:41:22.857242 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.857127    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn\" (UID: \"34ce7d00-e245-4eab-b9c9-6c9580000443\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"
Nov 03 19:41:22.857242 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.857212    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlq9s\" (UniqueName: \"kubernetes.io/projected/34ce7d00-e245-4eab-b9c9-6c9580000443-kube-api-access-xlq9s\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn\" (UID: \"34ce7d00-e245-4eab-b9c9-6c9580000443\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"
Nov 03 19:41:22.860411 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.860375    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"]
Nov 03 19:41:22.862825 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.862805    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-h7txl\""
Nov 03 19:41:22.955497 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.955448    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"]
Nov 03 19:41:22.958030 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.958006    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968\" (UID: \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"
Nov 03 19:41:22.958145 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.958047    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v78z\" (UniqueName: \"kubernetes.io/projected/dce9f81b-bc39-47ad-a93a-42adc8a3af59-kube-api-access-7v78z\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968\" (UID: \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"
Nov 03 19:41:22.958145 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.958130    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968\" (UID: \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"
Nov 03 19:41:22.958244 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.958225    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn\" (UID: \"34ce7d00-e245-4eab-b9c9-6c9580000443\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"
Nov 03 19:41:22.958299 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.958285    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn\" (UID: \"34ce7d00-e245-4eab-b9c9-6c9580000443\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"
Nov 03 19:41:22.958335 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.958310    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlq9s\" (UniqueName: \"kubernetes.io/projected/34ce7d00-e245-4eab-b9c9-6c9580000443-kube-api-access-xlq9s\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn\" (UID: \"34ce7d00-e245-4eab-b9c9-6c9580000443\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"
Nov 03 19:41:22.958549 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.958531    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn\" (UID: \"34ce7d00-e245-4eab-b9c9-6c9580000443\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"
Nov 03 19:41:22.958654 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.958636    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn\" (UID: \"34ce7d00-e245-4eab-b9c9-6c9580000443\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"
Nov 03 19:41:22.958782 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.958769    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"
Nov 03 19:41:22.962103 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.962086    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-v5d7l\""
Nov 03 19:41:22.970459 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.970440    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlq9s\" (UniqueName: \"kubernetes.io/projected/34ce7d00-e245-4eab-b9c9-6c9580000443-kube-api-access-xlq9s\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn\" (UID: \"34ce7d00-e245-4eab-b9c9-6c9580000443\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"
Nov 03 19:41:22.975542 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:22.975520    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"]
Nov 03 19:41:23.043189 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.043113    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"]
Nov 03 19:41:23.046354 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.046335    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"
Nov 03 19:41:23.050278 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.050260    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-wtdr6\""
Nov 03 19:41:23.056081 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.056058    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"]
Nov 03 19:41:23.058964 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.058935    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8f2j\" (UniqueName: \"kubernetes.io/projected/736dd8a0-4f61-4924-854b-bfc1091f3dc8-kube-api-access-c8f2j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p\" (UID: \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"
Nov 03 19:41:23.059072 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.058977    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p\" (UID: \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"
Nov 03 19:41:23.059072 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.059004    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p\" (UID: \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"
Nov 03 19:41:23.059186 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.059155    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968\" (UID: \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"
Nov 03 19:41:23.059244 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.059212    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v78z\" (UniqueName: \"kubernetes.io/projected/dce9f81b-bc39-47ad-a93a-42adc8a3af59-kube-api-access-7v78z\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968\" (UID: \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"
Nov 03 19:41:23.059244 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.059236    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968\" (UID: \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"
Nov 03 19:41:23.059613 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.059589    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968\" (UID: \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"
Nov 03 19:41:23.059709 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.059652    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968\" (UID: \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"
Nov 03 19:41:23.071868 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.071842    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v78z\" (UniqueName: \"kubernetes.io/projected/dce9f81b-bc39-47ad-a93a-42adc8a3af59-kube-api-access-7v78z\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968\" (UID: \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"
Nov 03 19:41:23.077690 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.077668    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"
Nov 03 19:41:23.160529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.159973    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8f2j\" (UniqueName: \"kubernetes.io/projected/736dd8a0-4f61-4924-854b-bfc1091f3dc8-kube-api-access-c8f2j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p\" (UID: \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"
Nov 03 19:41:23.160529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.160011    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p\" (UID: \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"
Nov 03 19:41:23.160529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.160030    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p\" (UID: \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"
Nov 03 19:41:23.160529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.160057    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8\" (UID: \"67562c00-1fce-4153-897f-758777b0033c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"
Nov 03 19:41:23.160529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.160078    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rtb5\" (UniqueName: \"kubernetes.io/projected/67562c00-1fce-4153-897f-758777b0033c-kube-api-access-9rtb5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8\" (UID: \"67562c00-1fce-4153-897f-758777b0033c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"
Nov 03 19:41:23.160529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.160122    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8\" (UID: \"67562c00-1fce-4153-897f-758777b0033c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"
Nov 03 19:41:23.160529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.160460    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p\" (UID: \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"
Nov 03 19:41:23.160847 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.160588    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p\" (UID: \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"
Nov 03 19:41:23.166001 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.165973    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"
Nov 03 19:41:23.172725 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.172700    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8f2j\" (UniqueName: \"kubernetes.io/projected/736dd8a0-4f61-4924-854b-bfc1091f3dc8-kube-api-access-c8f2j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p\" (UID: \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"
Nov 03 19:41:23.261019 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.260983    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8\" (UID: \"67562c00-1fce-4153-897f-758777b0033c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"
Nov 03 19:41:23.261199 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.261028    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rtb5\" (UniqueName: \"kubernetes.io/projected/67562c00-1fce-4153-897f-758777b0033c-kube-api-access-9rtb5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8\" (UID: \"67562c00-1fce-4153-897f-758777b0033c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"
Nov 03 19:41:23.261199 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.261085    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8\" (UID: \"67562c00-1fce-4153-897f-758777b0033c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"
Nov 03 19:41:23.261427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.261395    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8\" (UID: \"67562c00-1fce-4153-897f-758777b0033c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"
Nov 03 19:41:23.261529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.261437    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8\" (UID: \"67562c00-1fce-4153-897f-758777b0033c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"
Nov 03 19:41:23.273324 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.273295    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rtb5\" (UniqueName: \"kubernetes.io/projected/67562c00-1fce-4153-897f-758777b0033c-kube-api-access-9rtb5\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8\" (UID: \"67562c00-1fce-4153-897f-758777b0033c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"
Nov 03 19:41:23.274052 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.274037    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"
Nov 03 19:41:23.353156 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.353126    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"]
Nov 03 19:41:23.356513 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:41:23.356463    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ce7d00_e245_4eab_b9c9_6c9580000443.slice/crio-bcf722dad12f3935ce2074832c60c142a1c4bb918b035096b95d5921b11ade4f WatchSource:0}: Error finding container bcf722dad12f3935ce2074832c60c142a1c4bb918b035096b95d5921b11ade4f: Status 404 returned error can't find the container with id bcf722dad12f3935ce2074832c60c142a1c4bb918b035096b95d5921b11ade4f
Nov 03 19:41:23.363412 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.363387    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"
Nov 03 19:41:23.445999 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.445515    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"]
Nov 03 19:41:23.449986 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:41:23.449958    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce9f81b_bc39_47ad_a93a_42adc8a3af59.slice/crio-58d2dab5b44edeec816c8df0444b3145ae96eb7fdae95c5db199c47d0f64fa37 WatchSource:0}: Error finding container 58d2dab5b44edeec816c8df0444b3145ae96eb7fdae95c5db199c47d0f64fa37: Status 404 returned error can't find the container with id 58d2dab5b44edeec816c8df0444b3145ae96eb7fdae95c5db199c47d0f64fa37
Nov 03 19:41:23.547228 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.547197    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"]
Nov 03 19:41:23.550617 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:41:23.550585    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736dd8a0_4f61_4924_854b_bfc1091f3dc8.slice/crio-799fa89ac2222846e756ff1578f597111d4085c47a75eb0c0339b166245a67d2 WatchSource:0}: Error finding container 799fa89ac2222846e756ff1578f597111d4085c47a75eb0c0339b166245a67d2: Status 404 returned error can't find the container with id 799fa89ac2222846e756ff1578f597111d4085c47a75eb0c0339b166245a67d2
Nov 03 19:41:23.573295 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.573227    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"]
Nov 03 19:41:23.578211 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.578188    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.585441 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.585403    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 19:41:23.588229 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.588187    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 19:41:23.588324 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.588304    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 19:41:23.588964 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.588943    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-65hnm\""
Nov 03 19:41:23.592759 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.592739    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"]
Nov 03 19:41:23.638989 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.638957    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"]
Nov 03 19:41:23.641984 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:41:23.641959    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67562c00_1fce_4153_897f_758777b0033c.slice/crio-58395a3ea5a99b694c7e0fbe87d3540f7cd35213c27e23c65700d9a6a14ea901 WatchSource:0}: Error finding container 58395a3ea5a99b694c7e0fbe87d3540f7cd35213c27e23c65700d9a6a14ea901: Status 404 returned error can't find the container with id 58395a3ea5a99b694c7e0fbe87d3540f7cd35213c27e23c65700d9a6a14ea901
Nov 03 19:41:23.664283 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.664250    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.664410 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.664293    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.664410 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.664395    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.664524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.664452    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.664524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.664500    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.664645 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.664530    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:23.664645 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.664592    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bv47\" (UniqueName: \"kubernetes.io/projected/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-kube-api-access-6bv47\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.664645 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.664634    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.664794 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.664664    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.664794 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.664785    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.765752 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.765714    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.765935 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.765771    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.765935 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.765897    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.765935 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.765931    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.766167 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.765964    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.766167 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.766009    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bv47\" (UniqueName: \"kubernetes.io/projected/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-kube-api-access-6bv47\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.766167 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.766039    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.766167 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.766063    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.766167 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.766115    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.766415 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.766207    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.766415 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.766239    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.766415 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.766295    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.766592 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.766426    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.766832 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.766807    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.768276 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.768253    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.768953 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.768928    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.778702 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.778676    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bv47\" (UniqueName: \"kubernetes.io/projected/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-kube-api-access-6bv47\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.778702 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.778687    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fa9e0c2-1bfa-4b41-b033-8d286b4c63be-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tnfsk\" (UID: \"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:23.798183 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.798155    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:23.840575 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.840467    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968" event={"ID":"dce9f81b-bc39-47ad-a93a-42adc8a3af59","Type":"ContainerStarted","Data":"58d2dab5b44edeec816c8df0444b3145ae96eb7fdae95c5db199c47d0f64fa37"}
Nov 03 19:41:23.841809 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.841779    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn" event={"ID":"34ce7d00-e245-4eab-b9c9-6c9580000443","Type":"ContainerStarted","Data":"bcf722dad12f3935ce2074832c60c142a1c4bb918b035096b95d5921b11ade4f"}
Nov 03 19:41:23.843128 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.843094    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p" event={"ID":"736dd8a0-4f61-4924-854b-bfc1091f3dc8","Type":"ContainerStarted","Data":"799fa89ac2222846e756ff1578f597111d4085c47a75eb0c0339b166245a67d2"}
Nov 03 19:41:23.844508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.844378    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8" event={"ID":"67562c00-1fce-4153-897f-758777b0033c","Type":"ContainerStarted","Data":"58395a3ea5a99b694c7e0fbe87d3540f7cd35213c27e23c65700d9a6a14ea901"}
Nov 03 19:41:23.894994 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:23.894942    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:24.192609 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:24.192446    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"]
Nov 03 19:41:24.204605 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:41:24.204568    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa9e0c2_1bfa_4b41_b033_8d286b4c63be.slice/crio-9f40f7757cf7206a0ab339f22bb2fae6faefb5454cf6a3eda5715456f3911fd6 WatchSource:0}: Error finding container 9f40f7757cf7206a0ab339f22bb2fae6faefb5454cf6a3eda5715456f3911fd6: Status 404 returned error can't find the container with id 9f40f7757cf7206a0ab339f22bb2fae6faefb5454cf6a3eda5715456f3911fd6
Nov 03 19:41:24.851449 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:24.851364    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" event={"ID":"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be","Type":"ContainerStarted","Data":"9f40f7757cf7206a0ab339f22bb2fae6faefb5454cf6a3eda5715456f3911fd6"}
Nov 03 19:41:25.091923 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.090218    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"]
Nov 03 19:41:25.095558 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.094313    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"
Nov 03 19:41:25.100234 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.099829    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-fgw5t\""
Nov 03 19:41:25.104664 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.103567    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"]
Nov 03 19:41:25.183063 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.183026    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk\" (UID: \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"
Nov 03 19:41:25.183260 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.183098    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk\" (UID: \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"
Nov 03 19:41:25.183260 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.183154    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5vwc\" (UniqueName: \"kubernetes.io/projected/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-kube-api-access-f5vwc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk\" (UID: \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"
Nov 03 19:41:25.284249 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.284209    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk\" (UID: \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"
Nov 03 19:41:25.284434 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.284288    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5vwc\" (UniqueName: \"kubernetes.io/projected/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-kube-api-access-f5vwc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk\" (UID: \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"
Nov 03 19:41:25.284434 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.284372    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk\" (UID: \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"
Nov 03 19:41:25.284869 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.284844    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk\" (UID: \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"
Nov 03 19:41:25.284976 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.284905    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk\" (UID: \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"
Nov 03 19:41:25.301536 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.301500    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5vwc\" (UniqueName: \"kubernetes.io/projected/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-kube-api-access-f5vwc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk\" (UID: \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"
Nov 03 19:41:25.422148 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.421796    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"
Nov 03 19:41:25.751892 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.751774    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"]
Nov 03 19:41:25.771273 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:41:25.771234    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bed97f0_1734_41a9_ae3a_d52ae2d9724b.slice/crio-cf05e9b50864a45a0d9914e378057b7c98d92afb25239cea7db6da8583be0c68 WatchSource:0}: Error finding container cf05e9b50864a45a0d9914e378057b7c98d92afb25239cea7db6da8583be0c68: Status 404 returned error can't find the container with id cf05e9b50864a45a0d9914e378057b7c98d92afb25239cea7db6da8583be0c68
Nov 03 19:41:25.860597 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:25.860499    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk" event={"ID":"3bed97f0-1734-41a9-ae3a-d52ae2d9724b","Type":"ContainerStarted","Data":"cf05e9b50864a45a0d9914e378057b7c98d92afb25239cea7db6da8583be0c68"}
Nov 03 19:41:26.955403 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:26.955364    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:29.050939 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.049108    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"]
Nov 03 19:41:29.054679 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.054652    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"
Nov 03 19:41:29.066703 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.066674    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-6t477\""
Nov 03 19:41:29.067288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.067218    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"]
Nov 03 19:41:29.125805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.125535    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz\" (UID: \"b14e1df7-501b-45f5-a408-d1dd5a59f671\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"
Nov 03 19:41:29.125805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.125660    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz\" (UID: \"b14e1df7-501b-45f5-a408-d1dd5a59f671\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"
Nov 03 19:41:29.125805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.125695    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j78hv\" (UniqueName: \"kubernetes.io/projected/b14e1df7-501b-45f5-a408-d1dd5a59f671-kube-api-access-j78hv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz\" (UID: \"b14e1df7-501b-45f5-a408-d1dd5a59f671\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"
Nov 03 19:41:29.227379 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.226174    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz\" (UID: \"b14e1df7-501b-45f5-a408-d1dd5a59f671\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"
Nov 03 19:41:29.227379 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.226231    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78hv\" (UniqueName: \"kubernetes.io/projected/b14e1df7-501b-45f5-a408-d1dd5a59f671-kube-api-access-j78hv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz\" (UID: \"b14e1df7-501b-45f5-a408-d1dd5a59f671\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"
Nov 03 19:41:29.227379 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.226278    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz\" (UID: \"b14e1df7-501b-45f5-a408-d1dd5a59f671\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"
Nov 03 19:41:29.227379 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.226783    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz\" (UID: \"b14e1df7-501b-45f5-a408-d1dd5a59f671\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"
Nov 03 19:41:29.227379 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.227030    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz\" (UID: \"b14e1df7-501b-45f5-a408-d1dd5a59f671\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"
Nov 03 19:41:29.249265 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.249205    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j78hv\" (UniqueName: \"kubernetes.io/projected/b14e1df7-501b-45f5-a408-d1dd5a59f671-kube-api-access-j78hv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz\" (UID: \"b14e1df7-501b-45f5-a408-d1dd5a59f671\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"
Nov 03 19:41:29.378809 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.378617    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"
Nov 03 19:41:29.702716 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.702640    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"]
Nov 03 19:41:29.717629 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:41:29.717595    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14e1df7_501b_45f5_a408_d1dd5a59f671.slice/crio-e3acadf23009f52de54cebcc3213f552a78cd57051369c7d69073beaf2843d9d WatchSource:0}: Error finding container e3acadf23009f52de54cebcc3213f552a78cd57051369c7d69073beaf2843d9d: Status 404 returned error can't find the container with id e3acadf23009f52de54cebcc3213f552a78cd57051369c7d69073beaf2843d9d
Nov 03 19:41:29.891822 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:29.891789    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz" event={"ID":"b14e1df7-501b-45f5-a408-d1dd5a59f671","Type":"ContainerStarted","Data":"e3acadf23009f52de54cebcc3213f552a78cd57051369c7d69073beaf2843d9d"}
Nov 03 19:41:31.362770 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:31.362031    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:31.362770 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:31.362124    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:31.362770 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:31.362170    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:31.913195 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:31.912185    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" event={"ID":"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be","Type":"ContainerStarted","Data":"08ed40a9b89e859e2e90ae0e800a75bf5769201a5fab5266bbb512c909c0bb4b"}
Nov 03 19:41:32.897243 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:32.895789    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:32.902675 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:32.902224    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:32.929555 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:32.927842    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:32.929555 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:32.928384    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" podStartSLOduration=2.774135147 podStartE2EDuration="9.928373474s" totalImagesPullingTime="7.154238327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:31.94133178 +0000 UTC m=+244.096234210" watchObservedRunningTime="2026-11-03 19:41:32.928373474 +0000 UTC m=+245.083275890"
Nov 03 19:41:32.931768 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:32.931722    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:41:34.497986 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:34.497946    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:35.948510 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:35.948289    5659 generic.go:356] "Generic (PLEG): container finished" podID="05644f4e-db2b-48e8-a640-86c4e03fc043" containerID="4819c6bbc00337691b7fdbb5b580d1e170d58ff37c06c5e0180c5c713ce1f1a8" exitCode=0
Nov 03 19:41:35.948510 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:35.948399    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh" event={"ID":"05644f4e-db2b-48e8-a640-86c4e03fc043","Type":"ContainerDied","Data":"4819c6bbc00337691b7fdbb5b580d1e170d58ff37c06c5e0180c5c713ce1f1a8"}
Nov 03 19:41:37.960753 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:37.960719    5659 generic.go:356] "Generic (PLEG): container finished" podID="b14e1df7-501b-45f5-a408-d1dd5a59f671" containerID="55adeac00d48b1914e5b4f0824d1e8b798e5f450d1835b644abb968d9c0f5222" exitCode=0
Nov 03 19:41:37.961234 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:37.960772    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz" event={"ID":"b14e1df7-501b-45f5-a408-d1dd5a59f671","Type":"ContainerDied","Data":"55adeac00d48b1914e5b4f0824d1e8b798e5f450d1835b644abb968d9c0f5222"}
Nov 03 19:41:37.962531 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:37.962507    5659 generic.go:356] "Generic (PLEG): container finished" podID="dce9f81b-bc39-47ad-a93a-42adc8a3af59" containerID="c825d909910609322c8fe6111d4f6e1e3412cabded8f52920b0080e5569a6a0e" exitCode=0
Nov 03 19:41:37.966443 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:37.965318    5659 generic.go:356] "Generic (PLEG): container finished" podID="34ce7d00-e245-4eab-b9c9-6c9580000443" containerID="678ef76cebe8afb1d2d5eb1f5fb1e8b48806dcf4b57fc14894c04b6ab89805e2" exitCode=0
Nov 03 19:41:37.967085 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:37.966988    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968" event={"ID":"dce9f81b-bc39-47ad-a93a-42adc8a3af59","Type":"ContainerDied","Data":"c825d909910609322c8fe6111d4f6e1e3412cabded8f52920b0080e5569a6a0e"}
Nov 03 19:41:37.967085 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:37.967029    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn" event={"ID":"34ce7d00-e245-4eab-b9c9-6c9580000443","Type":"ContainerDied","Data":"678ef76cebe8afb1d2d5eb1f5fb1e8b48806dcf4b57fc14894c04b6ab89805e2"}
Nov 03 19:41:37.968004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:37.967978    5659 generic.go:356] "Generic (PLEG): container finished" podID="736dd8a0-4f61-4924-854b-bfc1091f3dc8" containerID="ea83de538b35a705ae8eb5f7bb850a54424a49524563d95e5b8120c979c8828c" exitCode=0
Nov 03 19:41:37.968118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:37.968073    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p" event={"ID":"736dd8a0-4f61-4924-854b-bfc1091f3dc8","Type":"ContainerDied","Data":"ea83de538b35a705ae8eb5f7bb850a54424a49524563d95e5b8120c979c8828c"}
Nov 03 19:41:37.970064 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:37.970035    5659 generic.go:356] "Generic (PLEG): container finished" podID="67562c00-1fce-4153-897f-758777b0033c" containerID="26a3255ca4945786ac8e6246a10349fd64dde454ab66998e7cdb1045372bc94b" exitCode=0
Nov 03 19:41:37.970158 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:37.970084    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8" event={"ID":"67562c00-1fce-4153-897f-758777b0033c","Type":"ContainerDied","Data":"26a3255ca4945786ac8e6246a10349fd64dde454ab66998e7cdb1045372bc94b"}
Nov 03 19:41:38.979260 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:38.979226    5659 generic.go:356] "Generic (PLEG): container finished" podID="3bed97f0-1734-41a9-ae3a-d52ae2d9724b" containerID="ef69671a298414687eb0e836958f29ab8456d312b6719faee6d26f40b0a804dc" exitCode=0
Nov 03 19:41:38.979736 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:38.979306    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk" event={"ID":"3bed97f0-1734-41a9-ae3a-d52ae2d9724b","Type":"ContainerDied","Data":"ef69671a298414687eb0e836958f29ab8456d312b6719faee6d26f40b0a804dc"}
Nov 03 19:41:38.981612 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:38.981589    5659 generic.go:356] "Generic (PLEG): container finished" podID="b14e1df7-501b-45f5-a408-d1dd5a59f671" containerID="496d9b1e57f8404913345d104e9c8e26a3229b203d84a8d3ecdad8721297b759" exitCode=0
Nov 03 19:41:38.981857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:38.981685    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz" event={"ID":"b14e1df7-501b-45f5-a408-d1dd5a59f671","Type":"ContainerDied","Data":"496d9b1e57f8404913345d104e9c8e26a3229b203d84a8d3ecdad8721297b759"}
Nov 03 19:41:38.983947 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:38.983916    5659 generic.go:356] "Generic (PLEG): container finished" podID="05644f4e-db2b-48e8-a640-86c4e03fc043" containerID="2a71d3ee332b08d83d22303947712874ef9e2cc26e4484c481e74f1fc8095580" exitCode=0
Nov 03 19:41:38.984015 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:38.983971    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh" event={"ID":"05644f4e-db2b-48e8-a640-86c4e03fc043","Type":"ContainerDied","Data":"2a71d3ee332b08d83d22303947712874ef9e2cc26e4484c481e74f1fc8095580"}
Nov 03 19:41:39.993227 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:39.993190    5659 generic.go:356] "Generic (PLEG): container finished" podID="67562c00-1fce-4153-897f-758777b0033c" containerID="ac0c395ad6d4eea12b695c7fa31898cbc7ad8d809bf3d10bfce9e3d75e172ec4" exitCode=0
Nov 03 19:41:39.993831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:39.993788    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8" event={"ID":"67562c00-1fce-4153-897f-758777b0033c","Type":"ContainerDied","Data":"ac0c395ad6d4eea12b695c7fa31898cbc7ad8d809bf3d10bfce9e3d75e172ec4"}
Nov 03 19:41:39.997724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:39.997696    5659 generic.go:356] "Generic (PLEG): container finished" podID="dce9f81b-bc39-47ad-a93a-42adc8a3af59" containerID="4649412d0e7d1b54a12f97216c9a281f432b37a9848b437a203928a6d2618d76" exitCode=0
Nov 03 19:41:39.997817 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:39.997796    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968" event={"ID":"dce9f81b-bc39-47ad-a93a-42adc8a3af59","Type":"ContainerDied","Data":"4649412d0e7d1b54a12f97216c9a281f432b37a9848b437a203928a6d2618d76"}
Nov 03 19:41:40.001737 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:40.001710    5659 generic.go:356] "Generic (PLEG): container finished" podID="34ce7d00-e245-4eab-b9c9-6c9580000443" containerID="1b440032aa90049607f33ffcb63a1b7942590ac00ab8abbd8d55304480734b75" exitCode=0
Nov 03 19:41:40.001882 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:40.001805    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn" event={"ID":"34ce7d00-e245-4eab-b9c9-6c9580000443","Type":"ContainerDied","Data":"1b440032aa90049607f33ffcb63a1b7942590ac00ab8abbd8d55304480734b75"}
Nov 03 19:41:40.006907 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:40.006881    5659 generic.go:356] "Generic (PLEG): container finished" podID="736dd8a0-4f61-4924-854b-bfc1091f3dc8" containerID="e30ee7615bb8631798123c7880c5a623bac26b366718240f18ddf3795cf831d8" exitCode=0
Nov 03 19:41:40.007013 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:40.006929    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p" event={"ID":"736dd8a0-4f61-4924-854b-bfc1091f3dc8","Type":"ContainerDied","Data":"e30ee7615bb8631798123c7880c5a623bac26b366718240f18ddf3795cf831d8"}
Nov 03 19:41:45.068550 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:45.067890    5659 generic.go:356] "Generic (PLEG): container finished" podID="3bed97f0-1734-41a9-ae3a-d52ae2d9724b" containerID="d35d7e5692151e769ffc28edb356746159a86573f3f3a5a7815064c38e39f3d9" exitCode=0
Nov 03 19:41:45.068550 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:45.068012    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk" event={"ID":"3bed97f0-1734-41a9-ae3a-d52ae2d9724b","Type":"ContainerDied","Data":"d35d7e5692151e769ffc28edb356746159a86573f3f3a5a7815064c38e39f3d9"}
Nov 03 19:41:45.233502 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:41:45.233440    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:41:51.290409 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:51.290372    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:51.415765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:51.415732    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:52.533276 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:52.532725    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk"]
Nov 03 19:41:52.585273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:52.585241    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk"]
Nov 03 19:41:52.585449 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:52.585369    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk"
Nov 03 19:41:52.590684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:52.590657    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 19:41:52.590684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:52.590675    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-kg4hj\""
Nov 03 19:41:52.590864 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:52.590722    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:41:52.680860 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:52.680790    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqvrl\" (UniqueName: \"kubernetes.io/projected/830e09d0-25bc-46c8-997f-13494acbe6ca-kube-api-access-qqvrl\") pod \"jobset-operator-674cd59dc5-9g4dk\" (UID: \"830e09d0-25bc-46c8-997f-13494acbe6ca\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk"
Nov 03 19:41:52.681146 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:52.680893    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/830e09d0-25bc-46c8-997f-13494acbe6ca-tmp\") pod \"jobset-operator-674cd59dc5-9g4dk\" (UID: \"830e09d0-25bc-46c8-997f-13494acbe6ca\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk"
Nov 03 19:41:52.782915 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:52.782068    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/830e09d0-25bc-46c8-997f-13494acbe6ca-tmp\") pod \"jobset-operator-674cd59dc5-9g4dk\" (UID: \"830e09d0-25bc-46c8-997f-13494acbe6ca\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk"
Nov 03 19:41:52.782915 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:52.782183    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqvrl\" (UniqueName: \"kubernetes.io/projected/830e09d0-25bc-46c8-997f-13494acbe6ca-kube-api-access-qqvrl\") pod \"jobset-operator-674cd59dc5-9g4dk\" (UID: \"830e09d0-25bc-46c8-997f-13494acbe6ca\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk"
Nov 03 19:41:52.782915 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:52.782813    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/830e09d0-25bc-46c8-997f-13494acbe6ca-tmp\") pod \"jobset-operator-674cd59dc5-9g4dk\" (UID: \"830e09d0-25bc-46c8-997f-13494acbe6ca\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk"
Nov 03 19:41:52.812374 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:52.812279    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqvrl\" (UniqueName: \"kubernetes.io/projected/830e09d0-25bc-46c8-997f-13494acbe6ca-kube-api-access-qqvrl\") pod \"jobset-operator-674cd59dc5-9g4dk\" (UID: \"830e09d0-25bc-46c8-997f-13494acbe6ca\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk"
Nov 03 19:41:52.918810 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:52.918564    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk"
Nov 03 19:41:53.287002 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:53.286970    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk"]
Nov 03 19:41:53.326869 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:41:53.326827    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod830e09d0_25bc_46c8_997f_13494acbe6ca.slice/crio-027be038f4ea4e4ce61415db286b15755e598dc3af3708407edf9b784f98abda WatchSource:0}: Error finding container 027be038f4ea4e4ce61415db286b15755e598dc3af3708407edf9b784f98abda: Status 404 returned error can't find the container with id 027be038f4ea4e4ce61415db286b15755e598dc3af3708407edf9b784f98abda
Nov 03 19:41:54.139825 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:54.139785    5659 generic.go:356] "Generic (PLEG): container finished" podID="b14e1df7-501b-45f5-a408-d1dd5a59f671" containerID="c124413e7d37addc0d260c9ce7e4c9c181ddbe7b135ebc03d3f859aad8496280" exitCode=0
Nov 03 19:41:54.140349 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:54.139888    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz" event={"ID":"b14e1df7-501b-45f5-a408-d1dd5a59f671","Type":"ContainerDied","Data":"c124413e7d37addc0d260c9ce7e4c9c181ddbe7b135ebc03d3f859aad8496280"}
Nov 03 19:41:54.144591 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:54.143264    5659 generic.go:356] "Generic (PLEG): container finished" podID="05644f4e-db2b-48e8-a640-86c4e03fc043" containerID="d3e46a6b7cc20134087ba852e9a0471f9ebd3200686a050e5d152e81a4c22094" exitCode=0
Nov 03 19:41:54.144591 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:54.143382    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh" event={"ID":"05644f4e-db2b-48e8-a640-86c4e03fc043","Type":"ContainerDied","Data":"d3e46a6b7cc20134087ba852e9a0471f9ebd3200686a050e5d152e81a4c22094"}
Nov 03 19:41:54.145373 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:54.145323    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk" event={"ID":"830e09d0-25bc-46c8-997f-13494acbe6ca","Type":"ContainerStarted","Data":"027be038f4ea4e4ce61415db286b15755e598dc3af3708407edf9b784f98abda"}
Nov 03 19:41:55.151861 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.151819    5659 generic.go:356] "Generic (PLEG): container finished" podID="dce9f81b-bc39-47ad-a93a-42adc8a3af59" containerID="b3e2fbabc2f5c91e5901b75b5c6b73702685b36c98135ca1a196181d82eb9b5a" exitCode=0
Nov 03 19:41:55.152348 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.151912    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968" event={"ID":"dce9f81b-bc39-47ad-a93a-42adc8a3af59","Type":"ContainerDied","Data":"b3e2fbabc2f5c91e5901b75b5c6b73702685b36c98135ca1a196181d82eb9b5a"}
Nov 03 19:41:55.156895 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.156555    5659 generic.go:356] "Generic (PLEG): container finished" podID="34ce7d00-e245-4eab-b9c9-6c9580000443" containerID="8abcb1453f19e9f3b4cb613f78df14742b52310cef54e3abc238082f02cb80a6" exitCode=0
Nov 03 19:41:55.156895 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.156673    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn" event={"ID":"34ce7d00-e245-4eab-b9c9-6c9580000443","Type":"ContainerDied","Data":"8abcb1453f19e9f3b4cb613f78df14742b52310cef54e3abc238082f02cb80a6"}
Nov 03 19:41:55.159341 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.159315    5659 generic.go:356] "Generic (PLEG): container finished" podID="736dd8a0-4f61-4924-854b-bfc1091f3dc8" containerID="1b8877db88d48b4b392f38fce8776c4481367e8fbce0e9f7c7a27cc528ebfbd0" exitCode=0
Nov 03 19:41:55.159464 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.159386    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p" event={"ID":"736dd8a0-4f61-4924-854b-bfc1091f3dc8","Type":"ContainerDied","Data":"1b8877db88d48b4b392f38fce8776c4481367e8fbce0e9f7c7a27cc528ebfbd0"}
Nov 03 19:41:55.161356 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.161333    5659 generic.go:356] "Generic (PLEG): container finished" podID="3bed97f0-1734-41a9-ae3a-d52ae2d9724b" containerID="5ee4eb47ab5152f45515c07864c6d339715ff954af784899954cea2da073f680" exitCode=0
Nov 03 19:41:55.161468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.161406    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk" event={"ID":"3bed97f0-1734-41a9-ae3a-d52ae2d9724b","Type":"ContainerDied","Data":"5ee4eb47ab5152f45515c07864c6d339715ff954af784899954cea2da073f680"}
Nov 03 19:41:55.163603 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.163580    5659 generic.go:356] "Generic (PLEG): container finished" podID="67562c00-1fce-4153-897f-758777b0033c" containerID="397908e378d6a8f6897a42e7b7c306b52a467a4289e094b9753b19aa8b90a610" exitCode=0
Nov 03 19:41:55.163705 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.163609    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8" event={"ID":"67562c00-1fce-4153-897f-758777b0033c","Type":"ContainerDied","Data":"397908e378d6a8f6897a42e7b7c306b52a467a4289e094b9753b19aa8b90a610"}
Nov 03 19:41:55.290140 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.290112    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"
Nov 03 19:41:55.329097 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.329066    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"
Nov 03 19:41:55.412781 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.412735    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-bundle\" (UniqueName: \"kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-bundle\") pod \"05644f4e-db2b-48e8-a640-86c4e03fc043\" (UID: \"05644f4e-db2b-48e8-a640-86c4e03fc043\") "
Nov 03 19:41:55.412970 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.412801    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14e1df7-501b-45f5-a408-d1dd5a59f671-kube-api-access-j78hv\" (UniqueName: \"kubernetes.io/projected/b14e1df7-501b-45f5-a408-d1dd5a59f671-kube-api-access-j78hv\") pod \"b14e1df7-501b-45f5-a408-d1dd5a59f671\" (UID: \"b14e1df7-501b-45f5-a408-d1dd5a59f671\") "
Nov 03 19:41:55.412970 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.412850    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-util\" (UniqueName: \"kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-util\") pod \"b14e1df7-501b-45f5-a408-d1dd5a59f671\" (UID: \"b14e1df7-501b-45f5-a408-d1dd5a59f671\") "
Nov 03 19:41:55.412970 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.412891    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-bundle\") pod \"b14e1df7-501b-45f5-a408-d1dd5a59f671\" (UID: \"b14e1df7-501b-45f5-a408-d1dd5a59f671\") "
Nov 03 19:41:55.413133 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.413000    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05644f4e-db2b-48e8-a640-86c4e03fc043-kube-api-access-rkw8s\" (UniqueName: \"kubernetes.io/projected/05644f4e-db2b-48e8-a640-86c4e03fc043-kube-api-access-rkw8s\") pod \"05644f4e-db2b-48e8-a640-86c4e03fc043\" (UID: \"05644f4e-db2b-48e8-a640-86c4e03fc043\") "
Nov 03 19:41:55.413133 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.413125    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-util\" (UniqueName: \"kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-util\") pod \"05644f4e-db2b-48e8-a640-86c4e03fc043\" (UID: \"05644f4e-db2b-48e8-a640-86c4e03fc043\") "
Nov 03 19:41:55.413225 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.413200    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-bundle" pod "05644f4e-db2b-48e8-a640-86c4e03fc043" (UID: "05644f4e-db2b-48e8-a640-86c4e03fc043"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:55.413417 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.413399    5659 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-bundle\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:55.413772 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.413746    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-bundle" pod "b14e1df7-501b-45f5-a408-d1dd5a59f671" (UID: "b14e1df7-501b-45f5-a408-d1dd5a59f671"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:55.415629 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.415604    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14e1df7-501b-45f5-a408-d1dd5a59f671-kube-api-access-j78hv" pod "b14e1df7-501b-45f5-a408-d1dd5a59f671" (UID: "b14e1df7-501b-45f5-a408-d1dd5a59f671"). InnerVolumeSpecName "kube-api-access-j78hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:41:55.415737 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.415629    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05644f4e-db2b-48e8-a640-86c4e03fc043-kube-api-access-rkw8s" pod "05644f4e-db2b-48e8-a640-86c4e03fc043" (UID: "05644f4e-db2b-48e8-a640-86c4e03fc043"). InnerVolumeSpecName "kube-api-access-rkw8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:41:55.417723 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.417687    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-util" pod "05644f4e-db2b-48e8-a640-86c4e03fc043" (UID: "05644f4e-db2b-48e8-a640-86c4e03fc043"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:55.418531 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.418507    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-util" pod "b14e1df7-501b-45f5-a408-d1dd5a59f671" (UID: "b14e1df7-501b-45f5-a408-d1dd5a59f671"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:55.514524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.514434    5659 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05644f4e-db2b-48e8-a640-86c4e03fc043-util\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:55.514524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.514466    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j78hv\" (UniqueName: \"kubernetes.io/projected/b14e1df7-501b-45f5-a408-d1dd5a59f671-kube-api-access-j78hv\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:55.514524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.514476    5659 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-util\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:55.514524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.514506    5659 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b14e1df7-501b-45f5-a408-d1dd5a59f671-bundle\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:55.514524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:55.514515    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkw8s\" (UniqueName: \"kubernetes.io/projected/05644f4e-db2b-48e8-a640-86c4e03fc043-kube-api-access-rkw8s\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.169684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.169610    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh"
Nov 03 19:41:56.169684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.169644    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644k8nzh" event={"ID":"05644f4e-db2b-48e8-a640-86c4e03fc043","Type":"ContainerDied","Data":"cc35217a5ebfdc5256c6e37d1529c2f784c17ae99f80d44231186a6e5f2b9242"}
Nov 03 19:41:56.169684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.169686    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc35217a5ebfdc5256c6e37d1529c2f784c17ae99f80d44231186a6e5f2b9242"
Nov 03 19:41:56.171573 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.171390    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk" event={"ID":"830e09d0-25bc-46c8-997f-13494acbe6ca","Type":"ContainerStarted","Data":"124183d2110e5d745719a18768ebdd9d960e4efba0cdcc4d70ea48c02dd02e50"}
Nov 03 19:41:56.173228 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.173185    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz" event={"ID":"b14e1df7-501b-45f5-a408-d1dd5a59f671","Type":"ContainerDied","Data":"e3acadf23009f52de54cebcc3213f552a78cd57051369c7d69073beaf2843d9d"}
Nov 03 19:41:56.173228 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.173210    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jmhz"
Nov 03 19:41:56.173364 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.173236    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3acadf23009f52de54cebcc3213f552a78cd57051369c7d69073beaf2843d9d"
Nov 03 19:41:56.331601 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.331568    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"
Nov 03 19:41:56.335076 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.335030    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g4dk" podStartSLOduration=1.976823482 podStartE2EDuration="4.335014038s" totalImagesPullingTime="2.358190556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:56.190613126 +0000 UTC m=+268.345515548" watchObservedRunningTime="2026-11-03 19:41:56.335014038 +0000 UTC m=+268.489916458"
Nov 03 19:41:56.381998 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.381968    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"
Nov 03 19:41:56.392123 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.392103    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"
Nov 03 19:41:56.397619 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.397599    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"
Nov 03 19:41:56.400281 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.400266    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"
Nov 03 19:41:56.428728 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.428645    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-bundle\") pod \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\" (UID: \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\") "
Nov 03 19:41:56.428728 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.428697    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-util\" (UniqueName: \"kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-util\") pod \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\" (UID: \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\") "
Nov 03 19:41:56.428938 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.428745    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce9f81b-bc39-47ad-a93a-42adc8a3af59-kube-api-access-7v78z\" (UniqueName: \"kubernetes.io/projected/dce9f81b-bc39-47ad-a93a-42adc8a3af59-kube-api-access-7v78z\") pod \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\" (UID: \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\") "
Nov 03 19:41:56.428938 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.428776    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-bundle\") pod \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\" (UID: \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\") "
Nov 03 19:41:56.428938 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.428825    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-kube-api-access-f5vwc\" (UniqueName: \"kubernetes.io/projected/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-kube-api-access-f5vwc\") pod \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\" (UID: \"3bed97f0-1734-41a9-ae3a-d52ae2d9724b\") "
Nov 03 19:41:56.428938 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.428855    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-util\" (UniqueName: \"kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-util\") pod \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\" (UID: \"dce9f81b-bc39-47ad-a93a-42adc8a3af59\") "
Nov 03 19:41:56.430249 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.430089    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-bundle" pod "dce9f81b-bc39-47ad-a93a-42adc8a3af59" (UID: "dce9f81b-bc39-47ad-a93a-42adc8a3af59"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:56.430249 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.430212    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-bundle" pod "3bed97f0-1734-41a9-ae3a-d52ae2d9724b" (UID: "3bed97f0-1734-41a9-ae3a-d52ae2d9724b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:56.432901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.432878    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce9f81b-bc39-47ad-a93a-42adc8a3af59-kube-api-access-7v78z" pod "dce9f81b-bc39-47ad-a93a-42adc8a3af59" (UID: "dce9f81b-bc39-47ad-a93a-42adc8a3af59"). InnerVolumeSpecName "kube-api-access-7v78z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:41:56.433381 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.433356    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-kube-api-access-f5vwc" pod "3bed97f0-1734-41a9-ae3a-d52ae2d9724b" (UID: "3bed97f0-1734-41a9-ae3a-d52ae2d9724b"). InnerVolumeSpecName "kube-api-access-f5vwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:41:56.434330 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.434310    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-util" pod "3bed97f0-1734-41a9-ae3a-d52ae2d9724b" (UID: "3bed97f0-1734-41a9-ae3a-d52ae2d9724b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:56.435987 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.435965    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-util" pod "dce9f81b-bc39-47ad-a93a-42adc8a3af59" (UID: "dce9f81b-bc39-47ad-a93a-42adc8a3af59"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:56.530290 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.530254    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-bundle\" (UniqueName: \"kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-bundle\") pod \"34ce7d00-e245-4eab-b9c9-6c9580000443\" (UID: \"34ce7d00-e245-4eab-b9c9-6c9580000443\") "
Nov 03 19:41:56.530290 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.530294    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-util\" (UniqueName: \"kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-util\") pod \"67562c00-1fce-4153-897f-758777b0033c\" (UID: \"67562c00-1fce-4153-897f-758777b0033c\") "
Nov 03 19:41:56.530561 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.530324    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67562c00-1fce-4153-897f-758777b0033c-kube-api-access-9rtb5\" (UniqueName: \"kubernetes.io/projected/67562c00-1fce-4153-897f-758777b0033c-kube-api-access-9rtb5\") pod \"67562c00-1fce-4153-897f-758777b0033c\" (UID: \"67562c00-1fce-4153-897f-758777b0033c\") "
Nov 03 19:41:56.530561 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.530381    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-util\" (UniqueName: \"kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-util\") pod \"34ce7d00-e245-4eab-b9c9-6c9580000443\" (UID: \"34ce7d00-e245-4eab-b9c9-6c9580000443\") "
Nov 03 19:41:56.530561 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.530403    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-bundle\") pod \"67562c00-1fce-4153-897f-758777b0033c\" (UID: \"67562c00-1fce-4153-897f-758777b0033c\") "
Nov 03 19:41:56.530561 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.530424    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ce7d00-e245-4eab-b9c9-6c9580000443-kube-api-access-xlq9s\" (UniqueName: \"kubernetes.io/projected/34ce7d00-e245-4eab-b9c9-6c9580000443-kube-api-access-xlq9s\") pod \"34ce7d00-e245-4eab-b9c9-6c9580000443\" (UID: \"34ce7d00-e245-4eab-b9c9-6c9580000443\") "
Nov 03 19:41:56.530561 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.530541    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736dd8a0-4f61-4924-854b-bfc1091f3dc8-kube-api-access-c8f2j\" (UniqueName: \"kubernetes.io/projected/736dd8a0-4f61-4924-854b-bfc1091f3dc8-kube-api-access-c8f2j\") pod \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\" (UID: \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\") "
Nov 03 19:41:56.530893 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.530863    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-bundle" pod "34ce7d00-e245-4eab-b9c9-6c9580000443" (UID: "34ce7d00-e245-4eab-b9c9-6c9580000443"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:56.530992 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.530931    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-bundle\") pod \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\" (UID: \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\") "
Nov 03 19:41:56.530992 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.530972    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-bundle" pod "67562c00-1fce-4153-897f-758777b0033c" (UID: "67562c00-1fce-4153-897f-758777b0033c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:56.531101 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.531004    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-util\" (UniqueName: \"kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-util\") pod \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\" (UID: \"736dd8a0-4f61-4924-854b-bfc1091f3dc8\") "
Nov 03 19:41:56.531365 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.531347    5659 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-bundle\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.531547 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.531532    5659 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-bundle\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.531658 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.531646    5659 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-util\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.531746 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.531733    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v78z\" (UniqueName: \"kubernetes.io/projected/dce9f81b-bc39-47ad-a93a-42adc8a3af59-kube-api-access-7v78z\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.531826 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.531801    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-bundle" pod "736dd8a0-4f61-4924-854b-bfc1091f3dc8" (UID: "736dd8a0-4f61-4924-854b-bfc1091f3dc8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:56.531879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.531815    5659 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-bundle\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.531879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.531861    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5vwc\" (UniqueName: \"kubernetes.io/projected/3bed97f0-1734-41a9-ae3a-d52ae2d9724b-kube-api-access-f5vwc\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.531984 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.531881    5659 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dce9f81b-bc39-47ad-a93a-42adc8a3af59-util\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.531984 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.531898    5659 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-bundle\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.533001 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.532977    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67562c00-1fce-4153-897f-758777b0033c-kube-api-access-9rtb5" pod "67562c00-1fce-4153-897f-758777b0033c" (UID: "67562c00-1fce-4153-897f-758777b0033c"). InnerVolumeSpecName "kube-api-access-9rtb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:41:56.533001 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.532991    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ce7d00-e245-4eab-b9c9-6c9580000443-kube-api-access-xlq9s" pod "34ce7d00-e245-4eab-b9c9-6c9580000443" (UID: "34ce7d00-e245-4eab-b9c9-6c9580000443"). InnerVolumeSpecName "kube-api-access-xlq9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:41:56.533284 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.533255    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736dd8a0-4f61-4924-854b-bfc1091f3dc8-kube-api-access-c8f2j" pod "736dd8a0-4f61-4924-854b-bfc1091f3dc8" (UID: "736dd8a0-4f61-4924-854b-bfc1091f3dc8"). InnerVolumeSpecName "kube-api-access-c8f2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:41:56.535172 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.535146    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-util" pod "67562c00-1fce-4153-897f-758777b0033c" (UID: "67562c00-1fce-4153-897f-758777b0033c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:56.535286 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.535268    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-util" pod "34ce7d00-e245-4eab-b9c9-6c9580000443" (UID: "34ce7d00-e245-4eab-b9c9-6c9580000443"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:56.537168 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.537147    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-util" pod "736dd8a0-4f61-4924-854b-bfc1091f3dc8" (UID: "736dd8a0-4f61-4924-854b-bfc1091f3dc8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:56.633021 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.632984    5659 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-bundle\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.633021 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.633012    5659 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/736dd8a0-4f61-4924-854b-bfc1091f3dc8-util\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.633021 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.633021    5659 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67562c00-1fce-4153-897f-758777b0033c-util\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.633021 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.633030    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rtb5\" (UniqueName: \"kubernetes.io/projected/67562c00-1fce-4153-897f-758777b0033c-kube-api-access-9rtb5\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.633284 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.633038    5659 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34ce7d00-e245-4eab-b9c9-6c9580000443-util\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.633284 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.633047    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlq9s\" (UniqueName: \"kubernetes.io/projected/34ce7d00-e245-4eab-b9c9-6c9580000443-kube-api-access-xlq9s\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:56.633284 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:56.633055    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8f2j\" (UniqueName: \"kubernetes.io/projected/736dd8a0-4f61-4924-854b-bfc1091f3dc8-kube-api-access-c8f2j\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:57.179214 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.179138    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn"
Nov 03 19:41:57.179650 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.179137    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s5xpn" event={"ID":"34ce7d00-e245-4eab-b9c9-6c9580000443","Type":"ContainerDied","Data":"bcf722dad12f3935ce2074832c60c142a1c4bb918b035096b95d5921b11ade4f"}
Nov 03 19:41:57.179650 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.179269    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcf722dad12f3935ce2074832c60c142a1c4bb918b035096b95d5921b11ade4f"
Nov 03 19:41:57.181153 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.181134    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p"
Nov 03 19:41:57.181259 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.181131    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clzw9p" event={"ID":"736dd8a0-4f61-4924-854b-bfc1091f3dc8","Type":"ContainerDied","Data":"799fa89ac2222846e756ff1578f597111d4085c47a75eb0c0339b166245a67d2"}
Nov 03 19:41:57.181259 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.181242    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="799fa89ac2222846e756ff1578f597111d4085c47a75eb0c0339b166245a67d2"
Nov 03 19:41:57.183119 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.183100    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk"
Nov 03 19:41:57.183232 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.183100    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6hfbk" event={"ID":"3bed97f0-1734-41a9-ae3a-d52ae2d9724b","Type":"ContainerDied","Data":"cf05e9b50864a45a0d9914e378057b7c98d92afb25239cea7db6da8583be0c68"}
Nov 03 19:41:57.183232 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.183206    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf05e9b50864a45a0d9914e378057b7c98d92afb25239cea7db6da8583be0c68"
Nov 03 19:41:57.185167 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.185139    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8" event={"ID":"67562c00-1fce-4153-897f-758777b0033c","Type":"ContainerDied","Data":"58395a3ea5a99b694c7e0fbe87d3540f7cd35213c27e23c65700d9a6a14ea901"}
Nov 03 19:41:57.185260 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.185186    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58395a3ea5a99b694c7e0fbe87d3540f7cd35213c27e23c65700d9a6a14ea901"
Nov 03 19:41:57.185260 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.185155    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75qth8"
Nov 03 19:41:57.187212 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.187192    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968"
Nov 03 19:41:57.187329 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.187219    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adj968" event={"ID":"dce9f81b-bc39-47ad-a93a-42adc8a3af59","Type":"ContainerDied","Data":"58d2dab5b44edeec816c8df0444b3145ae96eb7fdae95c5db199c47d0f64fa37"}
Nov 03 19:41:57.187329 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:57.187259    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58d2dab5b44edeec816c8df0444b3145ae96eb7fdae95c5db199c47d0f64fa37"
Nov 03 19:41:59.532790 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:59.532741    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:59.696456 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:41:59.696407    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:00.521184 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.521156    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:00.674468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674434    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"]
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674719    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b14e1df7-501b-45f5-a408-d1dd5a59f671" containerName="extract"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674730    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14e1df7-501b-45f5-a408-d1dd5a59f671" containerName="extract"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674743    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3bed97f0-1734-41a9-ae3a-d52ae2d9724b" containerName="extract"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674749    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bed97f0-1734-41a9-ae3a-d52ae2d9724b" containerName="extract"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674756    5659 cpu_manager.go:414] "containerMap: removing container" podUID="67562c00-1fce-4153-897f-758777b0033c" containerName="extract"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674761    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67562c00-1fce-4153-897f-758777b0033c" containerName="extract"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674768    5659 cpu_manager.go:414] "containerMap: removing container" podUID="736dd8a0-4f61-4924-854b-bfc1091f3dc8" containerName="util"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674773    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736dd8a0-4f61-4924-854b-bfc1091f3dc8" containerName="util"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674781    5659 cpu_manager.go:414] "containerMap: removing container" podUID="34ce7d00-e245-4eab-b9c9-6c9580000443" containerName="pull"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674786    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ce7d00-e245-4eab-b9c9-6c9580000443" containerName="pull"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674797    5659 cpu_manager.go:414] "containerMap: removing container" podUID="05644f4e-db2b-48e8-a640-86c4e03fc043" containerName="util"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674802    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05644f4e-db2b-48e8-a640-86c4e03fc043" containerName="util"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674809    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b14e1df7-501b-45f5-a408-d1dd5a59f671" containerName="util"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674813    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14e1df7-501b-45f5-a408-d1dd5a59f671" containerName="util"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674819    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b14e1df7-501b-45f5-a408-d1dd5a59f671" containerName="pull"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674823    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14e1df7-501b-45f5-a408-d1dd5a59f671" containerName="pull"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674829    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3bed97f0-1734-41a9-ae3a-d52ae2d9724b" containerName="pull"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674833    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bed97f0-1734-41a9-ae3a-d52ae2d9724b" containerName="pull"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674840    5659 cpu_manager.go:414] "containerMap: removing container" podUID="dce9f81b-bc39-47ad-a93a-42adc8a3af59" containerName="util"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674844    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce9f81b-bc39-47ad-a93a-42adc8a3af59" containerName="util"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674851    5659 cpu_manager.go:414] "containerMap: removing container" podUID="05644f4e-db2b-48e8-a640-86c4e03fc043" containerName="extract"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674855    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05644f4e-db2b-48e8-a640-86c4e03fc043" containerName="extract"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674861    5659 cpu_manager.go:414] "containerMap: removing container" podUID="67562c00-1fce-4153-897f-758777b0033c" containerName="util"
Nov 03 19:42:00.674857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674865    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67562c00-1fce-4153-897f-758777b0033c" containerName="util"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674873    5659 cpu_manager.go:414] "containerMap: removing container" podUID="dce9f81b-bc39-47ad-a93a-42adc8a3af59" containerName="pull"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674877    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce9f81b-bc39-47ad-a93a-42adc8a3af59" containerName="pull"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674882    5659 cpu_manager.go:414] "containerMap: removing container" podUID="dce9f81b-bc39-47ad-a93a-42adc8a3af59" containerName="extract"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674886    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce9f81b-bc39-47ad-a93a-42adc8a3af59" containerName="extract"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674893    5659 cpu_manager.go:414] "containerMap: removing container" podUID="34ce7d00-e245-4eab-b9c9-6c9580000443" containerName="util"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674897    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ce7d00-e245-4eab-b9c9-6c9580000443" containerName="util"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674905    5659 cpu_manager.go:414] "containerMap: removing container" podUID="05644f4e-db2b-48e8-a640-86c4e03fc043" containerName="pull"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674909    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05644f4e-db2b-48e8-a640-86c4e03fc043" containerName="pull"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674915    5659 cpu_manager.go:414] "containerMap: removing container" podUID="736dd8a0-4f61-4924-854b-bfc1091f3dc8" containerName="pull"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674919    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736dd8a0-4f61-4924-854b-bfc1091f3dc8" containerName="pull"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674926    5659 cpu_manager.go:414] "containerMap: removing container" podUID="736dd8a0-4f61-4924-854b-bfc1091f3dc8" containerName="extract"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674930    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736dd8a0-4f61-4924-854b-bfc1091f3dc8" containerName="extract"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674937    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3bed97f0-1734-41a9-ae3a-d52ae2d9724b" containerName="util"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674941    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bed97f0-1734-41a9-ae3a-d52ae2d9724b" containerName="util"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674947    5659 cpu_manager.go:414] "containerMap: removing container" podUID="67562c00-1fce-4153-897f-758777b0033c" containerName="pull"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674951    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67562c00-1fce-4153-897f-758777b0033c" containerName="pull"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674958    5659 cpu_manager.go:414] "containerMap: removing container" podUID="34ce7d00-e245-4eab-b9c9-6c9580000443" containerName="extract"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.674962    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ce7d00-e245-4eab-b9c9-6c9580000443" containerName="extract"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.675005    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ce7d00-e245-4eab-b9c9-6c9580000443" containerName="extract"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.675013    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b14e1df7-501b-45f5-a408-d1dd5a59f671" containerName="extract"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.675018    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bed97f0-1734-41a9-ae3a-d52ae2d9724b" containerName="extract"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.675023    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="67562c00-1fce-4153-897f-758777b0033c" containerName="extract"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.675030    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="05644f4e-db2b-48e8-a640-86c4e03fc043" containerName="extract"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.675035    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce9f81b-bc39-47ad-a93a-42adc8a3af59" containerName="extract"
Nov 03 19:42:00.675686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.675040    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="736dd8a0-4f61-4924-854b-bfc1091f3dc8" containerName="extract"
Nov 03 19:42:00.703712 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.703692    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.707667 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.707646    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"]
Nov 03 19:42:00.756503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.756457    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 19:42:00.756765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.756745    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 19:42:00.756829 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.756817    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-847xh\""
Nov 03 19:42:00.756829 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.756823    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 19:42:00.756913 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.756822    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 19:42:00.756913 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.756819    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 19:42:00.781253 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.781195    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4678c42e-c435-4d42-b40e-f75257db51aa-metrics-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.781253 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.781226    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4678c42e-c435-4d42-b40e-f75257db51aa-tmp\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.781397 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.781256    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5mkj\" (UniqueName: \"kubernetes.io/projected/4678c42e-c435-4d42-b40e-f75257db51aa-kube-api-access-f5mkj\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.781448 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.781409    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4678c42e-c435-4d42-b40e-f75257db51aa-manifests\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.781525 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.781471    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4678c42e-c435-4d42-b40e-f75257db51aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.781596 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.781575    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4678c42e-c435-4d42-b40e-f75257db51aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.848164 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.848135    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:00.882368 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.882341    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4678c42e-c435-4d42-b40e-f75257db51aa-metrics-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.882457 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.882372    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4678c42e-c435-4d42-b40e-f75257db51aa-tmp\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.882457 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.882404    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5mkj\" (UniqueName: \"kubernetes.io/projected/4678c42e-c435-4d42-b40e-f75257db51aa-kube-api-access-f5mkj\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.882457 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.882435    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4678c42e-c435-4d42-b40e-f75257db51aa-manifests\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.882656 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.882474    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4678c42e-c435-4d42-b40e-f75257db51aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.882656 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.882527    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4678c42e-c435-4d42-b40e-f75257db51aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.882926 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.882901    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4678c42e-c435-4d42-b40e-f75257db51aa-manifests\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.884760 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.884738    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4678c42e-c435-4d42-b40e-f75257db51aa-tmp\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.884983 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.884965    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4678c42e-c435-4d42-b40e-f75257db51aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.885055 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.884991    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4678c42e-c435-4d42-b40e-f75257db51aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.885114 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.885077    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4678c42e-c435-4d42-b40e-f75257db51aa-metrics-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.894742 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.894721    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5mkj\" (UniqueName: \"kubernetes.io/projected/4678c42e-c435-4d42-b40e-f75257db51aa-kube-api-access-f5mkj\") pod \"opendatahub-operator-controller-manager-8464c976fd-cpxbn\" (UID: \"4678c42e-c435-4d42-b40e-f75257db51aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:00.994741 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:00.994693    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:01.048496 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:01.048400    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:01.128719 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:01.128693    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:01.270014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:01.269981    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:01.359238 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:42:01.359211    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4678c42e_c435_4d42_b40e_f75257db51aa.slice/crio-253c64a8301d26783740b12e712365845f0f85d4344c97f12eed625c482fad80 WatchSource:0}: Error finding container 253c64a8301d26783740b12e712365845f0f85d4344c97f12eed625c482fad80: Status 404 returned error can't find the container with id 253c64a8301d26783740b12e712365845f0f85d4344c97f12eed625c482fad80
Nov 03 19:42:01.359357 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:01.359250    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"]
Nov 03 19:42:01.428023 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:01.427991    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:02.209024 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:02.208986    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn" event={"ID":"4678c42e-c435-4d42-b40e-f75257db51aa","Type":"ContainerStarted","Data":"253c64a8301d26783740b12e712365845f0f85d4344c97f12eed625c482fad80"}
Nov 03 19:42:03.339092 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:03.339055    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:05.163452 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:05.163412    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:06.230389 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:06.230348    5659 generic.go:356] "Generic (PLEG): container finished" podID="4678c42e-c435-4d42-b40e-f75257db51aa" containerID="c12efe6e4887b5b852757771de3d64cfef258fc79d1cf3886df3f679587b30bf" exitCode=0
Nov 03 19:42:06.230796 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:06.230412    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn" event={"ID":"4678c42e-c435-4d42-b40e-f75257db51aa","Type":"ContainerDied","Data":"c12efe6e4887b5b852757771de3d64cfef258fc79d1cf3886df3f679587b30bf"}
Nov 03 19:42:06.937006 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:06.936927    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:07.093885 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:07.093855    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:07.236983 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:07.236951    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn" event={"ID":"4678c42e-c435-4d42-b40e-f75257db51aa","Type":"ContainerStarted","Data":"e82ad78d9ef83842b55fa4b54b171e55e9d072c891baaa4c91867d90cc5b00a4"}
Nov 03 19:42:07.237458 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:07.237047    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:07.282321 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:07.282269    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn" podStartSLOduration=2.561317082 podStartE2EDuration="7.282255757s" totalImagesPullingTime="4.317451972s" totalInitContainerRuntime="403.486703ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:07.280630352 +0000 UTC m=+279.435532772" watchObservedRunningTime="2026-11-03 19:42:07.282255757 +0000 UTC m=+279.437158176"
Nov 03 19:42:07.541372 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:07.541295    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:08.572826 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:08.572792    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:09.160140 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:09.160098    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:10.615253 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:10.615226    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:10.776754 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:10.776721    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:12.783560 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:12.783525    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:14.373693 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.373656    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:14.690793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.690716    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"]
Nov 03 19:42:14.695746 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.695728    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"
Nov 03 19:42:14.703681 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.703660    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qd55\" (UniqueName: \"kubernetes.io/projected/06729d18-3d24-489f-9e57-4b24a6ca6323-kube-api-access-5qd55\") pod \"kuadrant-operator-controller-manager-678d87766f-l8fcr\" (UID: \"06729d18-3d24-489f-9e57-4b24a6ca6323\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"
Nov 03 19:42:14.703773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.703704    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06729d18-3d24-489f-9e57-4b24a6ca6323-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l8fcr\" (UID: \"06729d18-3d24-489f-9e57-4b24a6ca6323\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"
Nov 03 19:42:14.713023 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.713002    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 19:42:14.713023 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.713013    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 19:42:14.714460 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.714444    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-k488s\""
Nov 03 19:42:14.726131 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.726107    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:14.729998 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.729977    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"]
Nov 03 19:42:14.804774 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.804744    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qd55\" (UniqueName: \"kubernetes.io/projected/06729d18-3d24-489f-9e57-4b24a6ca6323-kube-api-access-5qd55\") pod \"kuadrant-operator-controller-manager-678d87766f-l8fcr\" (UID: \"06729d18-3d24-489f-9e57-4b24a6ca6323\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"
Nov 03 19:42:14.804944 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.804795    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06729d18-3d24-489f-9e57-4b24a6ca6323-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l8fcr\" (UID: \"06729d18-3d24-489f-9e57-4b24a6ca6323\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"
Nov 03 19:42:14.805216 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.805197    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06729d18-3d24-489f-9e57-4b24a6ca6323-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-l8fcr\" (UID: \"06729d18-3d24-489f-9e57-4b24a6ca6323\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"
Nov 03 19:42:14.824503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.824460    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qd55\" (UniqueName: \"kubernetes.io/projected/06729d18-3d24-489f-9e57-4b24a6ca6323-kube-api-access-5qd55\") pod \"kuadrant-operator-controller-manager-678d87766f-l8fcr\" (UID: \"06729d18-3d24-489f-9e57-4b24a6ca6323\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"
Nov 03 19:42:14.900440 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:14.900413    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:15.007843 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:15.007813    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"
Nov 03 19:42:15.043025 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:15.042989    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:15.214838 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:42:15.214803    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:42:15.294916 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:15.294894    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"]
Nov 03 19:42:15.296693 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:42:15.296658    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06729d18_3d24_489f_9e57_4b24a6ca6323.slice/crio-de56cb3cfa7a885b5d96d0a56a3f06975239318ee32b4e1ca35d20c6e6d829cb WatchSource:0}: Error finding container de56cb3cfa7a885b5d96d0a56a3f06975239318ee32b4e1ca35d20c6e6d829cb: Status 404 returned error can't find the container with id de56cb3cfa7a885b5d96d0a56a3f06975239318ee32b4e1ca35d20c6e6d829cb
Nov 03 19:42:15.730408 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:15.730378    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:15.963030 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:15.962995    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:16.275497 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:16.275456    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr" event={"ID":"06729d18-3d24-489f-9e57-4b24a6ca6323","Type":"ContainerStarted","Data":"de56cb3cfa7a885b5d96d0a56a3f06975239318ee32b4e1ca35d20c6e6d829cb"}
Nov 03 19:42:18.243514 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:18.243465    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-cpxbn"
Nov 03 19:42:18.712836 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:18.712798    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:20.297398 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.297355    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr" event={"ID":"06729d18-3d24-489f-9e57-4b24a6ca6323","Type":"ContainerStarted","Data":"a64053620fcb9a7cdc9f716ac7da3237387f0ec02695f6f8018ebae26d129ec2"}
Nov 03 19:42:20.297814 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.297516    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"
Nov 03 19:42:20.321593 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.321541    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr" podStartSLOduration=2.163595691 podStartE2EDuration="6.321524804s" totalImagesPullingTime="4.157929113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:20.320710531 +0000 UTC m=+292.475612950" watchObservedRunningTime="2026-11-03 19:42:20.321524804 +0000 UTC m=+292.476427222"
Nov 03 19:42:20.758668 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.758617    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"]
Nov 03 19:42:20.762382 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.762360    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"
Nov 03 19:42:20.773181 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.772532    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-579hd\""
Nov 03 19:42:20.773181 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.772580    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 19:42:20.773181 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.772684    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 19:42:20.774041 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.774021    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"]
Nov 03 19:42:20.820104 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.820073    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:20.860874 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.860844    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpzzs\" (UniqueName: \"kubernetes.io/projected/856603f3-24e3-4d9e-b6dc-68dc3f57d18a-kube-api-access-hpzzs\") pod \"kuadrant-console-plugin-66c4fb575d-t4sb5\" (UID: \"856603f3-24e3-4d9e-b6dc-68dc3f57d18a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"
Nov 03 19:42:20.861031 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.860898    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/856603f3-24e3-4d9e-b6dc-68dc3f57d18a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-t4sb5\" (UID: \"856603f3-24e3-4d9e-b6dc-68dc3f57d18a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"
Nov 03 19:42:20.861031 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.860984    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/856603f3-24e3-4d9e-b6dc-68dc3f57d18a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-t4sb5\" (UID: \"856603f3-24e3-4d9e-b6dc-68dc3f57d18a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"
Nov 03 19:42:20.962297 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.962263    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/856603f3-24e3-4d9e-b6dc-68dc3f57d18a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-t4sb5\" (UID: \"856603f3-24e3-4d9e-b6dc-68dc3f57d18a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"
Nov 03 19:42:20.962467 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.962306    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/856603f3-24e3-4d9e-b6dc-68dc3f57d18a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-t4sb5\" (UID: \"856603f3-24e3-4d9e-b6dc-68dc3f57d18a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"
Nov 03 19:42:20.962467 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.962355    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpzzs\" (UniqueName: \"kubernetes.io/projected/856603f3-24e3-4d9e-b6dc-68dc3f57d18a-kube-api-access-hpzzs\") pod \"kuadrant-console-plugin-66c4fb575d-t4sb5\" (UID: \"856603f3-24e3-4d9e-b6dc-68dc3f57d18a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"
Nov 03 19:42:20.963083 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.963053    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/856603f3-24e3-4d9e-b6dc-68dc3f57d18a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-t4sb5\" (UID: \"856603f3-24e3-4d9e-b6dc-68dc3f57d18a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"
Nov 03 19:42:20.964200 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.964176    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:20.964877 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.964858    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/856603f3-24e3-4d9e-b6dc-68dc3f57d18a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-t4sb5\" (UID: \"856603f3-24e3-4d9e-b6dc-68dc3f57d18a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"
Nov 03 19:42:20.974193 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:20.974171    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpzzs\" (UniqueName: \"kubernetes.io/projected/856603f3-24e3-4d9e-b6dc-68dc3f57d18a-kube-api-access-hpzzs\") pod \"kuadrant-console-plugin-66c4fb575d-t4sb5\" (UID: \"856603f3-24e3-4d9e-b6dc-68dc3f57d18a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"
Nov 03 19:42:21.074646 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:21.074565    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"
Nov 03 19:42:21.190005 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:21.189969    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:21.354877 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:21.354825    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5"]
Nov 03 19:42:21.356848 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:42:21.356817    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod856603f3_24e3_4d9e_b6dc_68dc3f57d18a.slice/crio-c4565df21528efd6d2608e766e06a52cfce4e7445fe6bf87384b1d43e98d5bab WatchSource:0}: Error finding container c4565df21528efd6d2608e766e06a52cfce4e7445fe6bf87384b1d43e98d5bab: Status 404 returned error can't find the container with id c4565df21528efd6d2608e766e06a52cfce4e7445fe6bf87384b1d43e98d5bab
Nov 03 19:42:22.310010 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:22.309968    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5" event={"ID":"856603f3-24e3-4d9e-b6dc-68dc3f57d18a","Type":"ContainerStarted","Data":"c4565df21528efd6d2608e766e06a52cfce4e7445fe6bf87384b1d43e98d5bab"}
Nov 03 19:42:24.031047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:24.031013    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:24.236834 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:24.236796    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:25.791660 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:25.791630    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:25.967469 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:25.967440    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:26.218747 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.218711    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:26.331905 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.331867    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5" event={"ID":"856603f3-24e3-4d9e-b6dc-68dc3f57d18a","Type":"ContainerStarted","Data":"8bfb2919bfcaf7df314afbcbacf42dbf42a95c00a824c437a86ac9c26055171d"}
Nov 03 19:42:26.353409 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.353361    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-t4sb5" podStartSLOduration=1.776414104 podStartE2EDuration="6.353348626s" totalImagesPullingTime="4.576934522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:26.352177449 +0000 UTC m=+298.507079868" watchObservedRunningTime="2026-11-03 19:42:26.353348626 +0000 UTC m=+298.508251045"
Nov 03 19:42:26.483002 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.482924    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc"]
Nov 03 19:42:26.486554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.486538    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc"
Nov 03 19:42:26.501427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.501398    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc"]
Nov 03 19:42:26.502697 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.502675    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 19:42:26.502814 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.502727    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 19:42:26.503100 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.503084    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-pbsrg\""
Nov 03 19:42:26.515417 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.515359    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxlh\" (UniqueName: \"kubernetes.io/projected/9d882201-bac3-4da8-abb1-8cb1a8edefef-kube-api-access-cvxlh\") pod \"cert-manager-webhook-5dbbbf75cc-59rwc\" (UID: \"9d882201-bac3-4da8-abb1-8cb1a8edefef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc"
Nov 03 19:42:26.515604 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.515554    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d882201-bac3-4da8-abb1-8cb1a8edefef-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-59rwc\" (UID: \"9d882201-bac3-4da8-abb1-8cb1a8edefef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc"
Nov 03 19:42:26.616619 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.616585    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxlh\" (UniqueName: \"kubernetes.io/projected/9d882201-bac3-4da8-abb1-8cb1a8edefef-kube-api-access-cvxlh\") pod \"cert-manager-webhook-5dbbbf75cc-59rwc\" (UID: \"9d882201-bac3-4da8-abb1-8cb1a8edefef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc"
Nov 03 19:42:26.616782 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.616660    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d882201-bac3-4da8-abb1-8cb1a8edefef-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-59rwc\" (UID: \"9d882201-bac3-4da8-abb1-8cb1a8edefef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc"
Nov 03 19:42:26.634386 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.634356    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d882201-bac3-4da8-abb1-8cb1a8edefef-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-59rwc\" (UID: \"9d882201-bac3-4da8-abb1-8cb1a8edefef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc"
Nov 03 19:42:26.634520 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.634425    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxlh\" (UniqueName: \"kubernetes.io/projected/9d882201-bac3-4da8-abb1-8cb1a8edefef-kube-api-access-cvxlh\") pod \"cert-manager-webhook-5dbbbf75cc-59rwc\" (UID: \"9d882201-bac3-4da8-abb1-8cb1a8edefef\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc"
Nov 03 19:42:26.722559 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.722520    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:26.797432 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:26.797346    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc"
Nov 03 19:42:27.079278 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:27.079079    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc"]
Nov 03 19:42:27.081661 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:42:27.081631    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d882201_bac3_4da8_abb1_8cb1a8edefef.slice/crio-15fdb0b4b13f4a50b67b3b153b3d2ec1afa940e685ec366fa8b9b69548a9c3b9 WatchSource:0}: Error finding container 15fdb0b4b13f4a50b67b3b153b3d2ec1afa940e685ec366fa8b9b69548a9c3b9: Status 404 returned error can't find the container with id 15fdb0b4b13f4a50b67b3b153b3d2ec1afa940e685ec366fa8b9b69548a9c3b9
Nov 03 19:42:27.154775 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:27.154721    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:27.337088 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:27.336996    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc" event={"ID":"9d882201-bac3-4da8-abb1-8cb1a8edefef","Type":"ContainerStarted","Data":"15fdb0b4b13f4a50b67b3b153b3d2ec1afa940e685ec366fa8b9b69548a9c3b9"}
Nov 03 19:42:27.380496 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:27.380455    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:30.353008 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:30.352975    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc" event={"ID":"9d882201-bac3-4da8-abb1-8cb1a8edefef","Type":"ContainerStarted","Data":"069938c441f6f727f01f799e10c5ea614e9b95ebe0e8d513b9b876ddfc41368e"}
Nov 03 19:42:30.353387 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:30.353107    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc"
Nov 03 19:42:30.369443 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:30.369416    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:30.372680 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:30.372635    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc" podStartSLOduration=1.6591438219999999 podStartE2EDuration="4.372622765s" totalImagesPullingTime="2.713478943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:30.37125501 +0000 UTC m=+302.526157430" watchObservedRunningTime="2026-11-03 19:42:30.372622765 +0000 UTC m=+302.527525184"
Nov 03 19:42:30.537866 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:30.537833    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:31.016920 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:31.016885    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:31.210393 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:31.210361    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:31.304280 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:31.304213    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"
Nov 03 19:42:31.391141 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:31.391108    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:31.623392 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:31.623306    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:34.241071 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:34.241038    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:34.598874 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:34.598786    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:35.846008 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:35.845971    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-c8rm9"]
Nov 03 19:42:35.849407 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:35.849390    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-c8rm9"
Nov 03 19:42:35.859564 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:35.859540    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7bxh7\""
Nov 03 19:42:35.860668 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:35.860647    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-c8rm9"]
Nov 03 19:42:36.001347 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:36.001309    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ea1b9f2-228b-4a8b-a44a-ec729c338d66-bound-sa-token\") pod \"cert-manager-597b95cb84-c8rm9\" (UID: \"1ea1b9f2-228b-4a8b-a44a-ec729c338d66\") " pod="cert-manager/cert-manager-597b95cb84-c8rm9"
Nov 03 19:42:36.001347 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:36.001347    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wb46\" (UniqueName: \"kubernetes.io/projected/1ea1b9f2-228b-4a8b-a44a-ec729c338d66-kube-api-access-4wb46\") pod \"cert-manager-597b95cb84-c8rm9\" (UID: \"1ea1b9f2-228b-4a8b-a44a-ec729c338d66\") " pod="cert-manager/cert-manager-597b95cb84-c8rm9"
Nov 03 19:42:36.102217 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:36.102120    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ea1b9f2-228b-4a8b-a44a-ec729c338d66-bound-sa-token\") pod \"cert-manager-597b95cb84-c8rm9\" (UID: \"1ea1b9f2-228b-4a8b-a44a-ec729c338d66\") " pod="cert-manager/cert-manager-597b95cb84-c8rm9"
Nov 03 19:42:36.102217 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:36.102171    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wb46\" (UniqueName: \"kubernetes.io/projected/1ea1b9f2-228b-4a8b-a44a-ec729c338d66-kube-api-access-4wb46\") pod \"cert-manager-597b95cb84-c8rm9\" (UID: \"1ea1b9f2-228b-4a8b-a44a-ec729c338d66\") " pod="cert-manager/cert-manager-597b95cb84-c8rm9"
Nov 03 19:42:36.117321 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:36.117288    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ea1b9f2-228b-4a8b-a44a-ec729c338d66-bound-sa-token\") pod \"cert-manager-597b95cb84-c8rm9\" (UID: \"1ea1b9f2-228b-4a8b-a44a-ec729c338d66\") " pod="cert-manager/cert-manager-597b95cb84-c8rm9"
Nov 03 19:42:36.117471 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:36.117451    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wb46\" (UniqueName: \"kubernetes.io/projected/1ea1b9f2-228b-4a8b-a44a-ec729c338d66-kube-api-access-4wb46\") pod \"cert-manager-597b95cb84-c8rm9\" (UID: \"1ea1b9f2-228b-4a8b-a44a-ec729c338d66\") " pod="cert-manager/cert-manager-597b95cb84-c8rm9"
Nov 03 19:42:36.160395 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:36.160359    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-c8rm9"
Nov 03 19:42:36.360965 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:36.360880    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-59rwc"
Nov 03 19:42:36.433030 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:36.433002    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-c8rm9"]
Nov 03 19:42:36.436314 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:42:36.436277    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ea1b9f2_228b_4a8b_a44a_ec729c338d66.slice/crio-a71e6e058c8b4caf1fdf55149e611f4aac3159e139e9004e47c1cb7ed70244c7 WatchSource:0}: Error finding container a71e6e058c8b4caf1fdf55149e611f4aac3159e139e9004e47c1cb7ed70244c7: Status 404 returned error can't find the container with id a71e6e058c8b4caf1fdf55149e611f4aac3159e139e9004e47c1cb7ed70244c7
Nov 03 19:42:37.386668 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:37.386630    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-c8rm9" event={"ID":"1ea1b9f2-228b-4a8b-a44a-ec729c338d66","Type":"ContainerStarted","Data":"ccbab9bf89988dbaa368d49eb5d7b7103d92f3886e4c67148b3d5e672fe1c1e6"}
Nov 03 19:42:37.387039 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:37.386677    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-c8rm9" event={"ID":"1ea1b9f2-228b-4a8b-a44a-ec729c338d66","Type":"ContainerStarted","Data":"a71e6e058c8b4caf1fdf55149e611f4aac3159e139e9004e47c1cb7ed70244c7"}
Nov 03 19:42:37.415472 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:37.415425    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-c8rm9" podStartSLOduration=2.415412329 podStartE2EDuration="2.415412329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:37.414165034 +0000 UTC m=+309.569067455" watchObservedRunningTime="2026-11-03 19:42:37.415412329 +0000 UTC m=+309.570314747"
Nov 03 19:42:37.604804 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:37.604767    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:37.809341 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:37.809308    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:37.927431 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:37.927391    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"]
Nov 03 19:42:37.931009 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:37.930987    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"
Nov 03 19:42:37.973539 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:37.973504    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"]
Nov 03 19:42:37.992828 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:37.992795    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 19:42:37.992977 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:37.992917    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fkx6r\""
Nov 03 19:42:37.992977 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:37.992945    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 19:42:38.018995 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.018964    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/727ca680-846f-4226-b4ee-1dad5f785224-tmp\") pod \"kube-auth-proxy-79ddfb4884-vtwjp\" (UID: \"727ca680-846f-4226-b4ee-1dad5f785224\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"
Nov 03 19:42:38.019145 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.019017    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727ca680-846f-4226-b4ee-1dad5f785224-tls-certs\") pod \"kube-auth-proxy-79ddfb4884-vtwjp\" (UID: \"727ca680-846f-4226-b4ee-1dad5f785224\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"
Nov 03 19:42:38.019145 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.019119    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkqpt\" (UniqueName: \"kubernetes.io/projected/727ca680-846f-4226-b4ee-1dad5f785224-kube-api-access-bkqpt\") pod \"kube-auth-proxy-79ddfb4884-vtwjp\" (UID: \"727ca680-846f-4226-b4ee-1dad5f785224\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"
Nov 03 19:42:38.056131 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.056098    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:38.120334 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.120253    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727ca680-846f-4226-b4ee-1dad5f785224-tls-certs\") pod \"kube-auth-proxy-79ddfb4884-vtwjp\" (UID: \"727ca680-846f-4226-b4ee-1dad5f785224\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"
Nov 03 19:42:38.120334 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.120301    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkqpt\" (UniqueName: \"kubernetes.io/projected/727ca680-846f-4226-b4ee-1dad5f785224-kube-api-access-bkqpt\") pod \"kube-auth-proxy-79ddfb4884-vtwjp\" (UID: \"727ca680-846f-4226-b4ee-1dad5f785224\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"
Nov 03 19:42:38.120607 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.120345    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/727ca680-846f-4226-b4ee-1dad5f785224-tmp\") pod \"kube-auth-proxy-79ddfb4884-vtwjp\" (UID: \"727ca680-846f-4226-b4ee-1dad5f785224\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"
Nov 03 19:42:38.122759 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.122729    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/727ca680-846f-4226-b4ee-1dad5f785224-tmp\") pod \"kube-auth-proxy-79ddfb4884-vtwjp\" (UID: \"727ca680-846f-4226-b4ee-1dad5f785224\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"
Nov 03 19:42:38.123253 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.123228    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727ca680-846f-4226-b4ee-1dad5f785224-tls-certs\") pod \"kube-auth-proxy-79ddfb4884-vtwjp\" (UID: \"727ca680-846f-4226-b4ee-1dad5f785224\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"
Nov 03 19:42:38.142079 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.142048    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkqpt\" (UniqueName: \"kubernetes.io/projected/727ca680-846f-4226-b4ee-1dad5f785224-kube-api-access-bkqpt\") pod \"kube-auth-proxy-79ddfb4884-vtwjp\" (UID: \"727ca680-846f-4226-b4ee-1dad5f785224\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"
Nov 03 19:42:38.245615 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.245589    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"
Nov 03 19:42:38.396646 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.396532    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:38.532291 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.532261    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp"]
Nov 03 19:42:38.535501 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:42:38.535463    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727ca680_846f_4226_b4ee_1dad5f785224.slice/crio-eb4e5c137ef204b87074a4db8676334d749705684dfed6f24ce3f6596f824dc3 WatchSource:0}: Error finding container eb4e5c137ef204b87074a4db8676334d749705684dfed6f24ce3f6596f824dc3: Status 404 returned error can't find the container with id eb4e5c137ef204b87074a4db8676334d749705684dfed6f24ce3f6596f824dc3
Nov 03 19:42:38.604921 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.604853    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:38.769567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:38.769522    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:39.030549 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:39.030458    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:39.256812 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:39.256772    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:39.398605 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:39.398520    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp" event={"ID":"727ca680-846f-4226-b4ee-1dad5f785224","Type":"ContainerStarted","Data":"eb4e5c137ef204b87074a4db8676334d749705684dfed6f24ce3f6596f824dc3"}
Nov 03 19:42:40.084629 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:40.084553    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:40.310956 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:40.310897    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:41.410626 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.410584    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp" event={"ID":"727ca680-846f-4226-b4ee-1dad5f785224","Type":"ContainerStarted","Data":"b41ff770d596bbeb8cecf228efbf8870c2ddd5f07b7b1f80e1bc235747db17c2"}
Nov 03 19:42:41.431312 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.431259    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79ddfb4884-vtwjp" podStartSLOduration=1.751581248 podStartE2EDuration="4.431245429s" totalImagesPullingTime="2.679664181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:41.430167441 +0000 UTC m=+313.585069860" watchObservedRunningTime="2026-11-03 19:42:41.431245429 +0000 UTC m=+313.586147848"
Nov 03 19:42:41.451997 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.451969    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:41.615919 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.615847    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:41.865314 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.865277    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"]
Nov 03 19:42:41.868920 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.868870    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:41.882814 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.882786    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 19:42:41.882940 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.882793    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 19:42:41.885591 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.885571    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-r94rc\""
Nov 03 19:42:41.885710 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.885694    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 03 19:42:41.892665 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.892643    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"]
Nov 03 19:42:41.955476 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.955440    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aa079182-95a7-4454-be04-45f06a41088f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fxxs9\" (UID: \"aa079182-95a7-4454-be04-45f06a41088f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:41.955666 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.955589    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa079182-95a7-4454-be04-45f06a41088f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fxxs9\" (UID: \"aa079182-95a7-4454-be04-45f06a41088f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:41.955666 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.955617    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa079182-95a7-4454-be04-45f06a41088f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fxxs9\" (UID: \"aa079182-95a7-4454-be04-45f06a41088f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:41.955666 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:41.955640    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcxmq\" (UniqueName: \"kubernetes.io/projected/aa079182-95a7-4454-be04-45f06a41088f-kube-api-access-kcxmq\") pod \"kserve-module-controller-manager-7566cf5fc8-fxxs9\" (UID: \"aa079182-95a7-4454-be04-45f06a41088f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:42.056399 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:42.056367    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa079182-95a7-4454-be04-45f06a41088f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fxxs9\" (UID: \"aa079182-95a7-4454-be04-45f06a41088f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:42.056599 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:42.056406    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa079182-95a7-4454-be04-45f06a41088f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fxxs9\" (UID: \"aa079182-95a7-4454-be04-45f06a41088f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:42.056599 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:42.056430    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcxmq\" (UniqueName: \"kubernetes.io/projected/aa079182-95a7-4454-be04-45f06a41088f-kube-api-access-kcxmq\") pod \"kserve-module-controller-manager-7566cf5fc8-fxxs9\" (UID: \"aa079182-95a7-4454-be04-45f06a41088f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:42.056599 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:42.056560    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aa079182-95a7-4454-be04-45f06a41088f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fxxs9\" (UID: \"aa079182-95a7-4454-be04-45f06a41088f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:42.056815 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:42.056796    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa079182-95a7-4454-be04-45f06a41088f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fxxs9\" (UID: \"aa079182-95a7-4454-be04-45f06a41088f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:42.056892 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:42.056873    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aa079182-95a7-4454-be04-45f06a41088f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fxxs9\" (UID: \"aa079182-95a7-4454-be04-45f06a41088f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:42.057322 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:42.057304    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa079182-95a7-4454-be04-45f06a41088f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fxxs9\" (UID: \"aa079182-95a7-4454-be04-45f06a41088f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:42.069283 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:42.069259    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcxmq\" (UniqueName: \"kubernetes.io/projected/aa079182-95a7-4454-be04-45f06a41088f-kube-api-access-kcxmq\") pod \"kserve-module-controller-manager-7566cf5fc8-fxxs9\" (UID: \"aa079182-95a7-4454-be04-45f06a41088f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:42.186541 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:42.186450    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:42.463099 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:42.463011    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"]
Nov 03 19:42:42.465186 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:42:42.465159    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa079182_95a7_4454_be04_45f06a41088f.slice/crio-65934a4df1ee205dbd9cb606427706a3d52e31fa9b54aa1c2ef66ba042ae38bd WatchSource:0}: Error finding container 65934a4df1ee205dbd9cb606427706a3d52e31fa9b54aa1c2ef66ba042ae38bd: Status 404 returned error can't find the container with id 65934a4df1ee205dbd9cb606427706a3d52e31fa9b54aa1c2ef66ba042ae38bd
Nov 03 19:42:42.689744 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:42.689708    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:43.420618 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:43.420579    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9" event={"ID":"aa079182-95a7-4454-be04-45f06a41088f","Type":"ContainerStarted","Data":"65934a4df1ee205dbd9cb606427706a3d52e31fa9b54aa1c2ef66ba042ae38bd"}
Nov 03 19:42:45.216102 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:42:45.216066    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:42:45.432215 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:45.432160    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9" event={"ID":"aa079182-95a7-4454-be04-45f06a41088f","Type":"ContainerStarted","Data":"0e1baf8296ba31b729a60d49f86ce1a5b5b6ef208c22c417d870e83f733d4c30"}
Nov 03 19:42:45.432438 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:45.432269    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:42:45.456232 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:45.456185    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9" podStartSLOduration=2.277690968 podStartE2EDuration="4.456172214s" totalImagesPullingTime="2.178481246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:45.45484211 +0000 UTC m=+317.609744529" watchObservedRunningTime="2026-11-03 19:42:45.456172214 +0000 UTC m=+317.611074632"
Nov 03 19:42:45.781258 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:45.781222    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:45.936516 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:45.936467    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:47.679008 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:47.678975    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:47.815983 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:47.815954    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:49.354394 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:49.354359    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:49.484429 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:49.484397    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:49.788082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:49.788048    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:49.962781 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:49.962731    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:50.120072 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:50.119979    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:58.785407 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:58.785374    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:59.065996 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:59.065882    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:59.209740 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:42:59.209696    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:00.456640 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:00.456608    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:00.666425 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:00.666347    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:00.919404 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:00.919254    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:01.132417 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:01.132346    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:01.460478 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:01.460437    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:03.005762 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:03.005708    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:03.171906 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:03.171873    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:06.437838 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:06.437801    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fxxs9"
Nov 03 19:43:15.216339 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:43:15.216292    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:43:20.976969 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:20.976867    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:21.161548 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:21.161494    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:21.308197 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:21.308086    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:25.440706 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:25.440652    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:25.585408 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:25.585350    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:26.437713 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.437678    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"]
Nov 03 19:43:26.444104 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.444079    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:26.453534 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.452125    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"]
Nov 03 19:43:26.456698 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.456669    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 19:43:26.456851 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.456832    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 19:43:26.465127 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.465094    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 19:43:26.465127 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.465123    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-xb8l2\""
Nov 03 19:43:26.550682 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.550643    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/67be1eda-e12d-4a81-9785-5c5c8c3989d2-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-564sz\" (UID: \"67be1eda-e12d-4a81-9785-5c5c8c3989d2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:26.550858 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.550711    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67be1eda-e12d-4a81-9785-5c5c8c3989d2-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-564sz\" (UID: \"67be1eda-e12d-4a81-9785-5c5c8c3989d2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:26.550858 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.550817    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddpxt\" (UniqueName: \"kubernetes.io/projected/67be1eda-e12d-4a81-9785-5c5c8c3989d2-kube-api-access-ddpxt\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-564sz\" (UID: \"67be1eda-e12d-4a81-9785-5c5c8c3989d2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:26.550858 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.550854    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67be1eda-e12d-4a81-9785-5c5c8c3989d2-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-564sz\" (UID: \"67be1eda-e12d-4a81-9785-5c5c8c3989d2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:26.559404 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.559374    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:26.651895 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.651860    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67be1eda-e12d-4a81-9785-5c5c8c3989d2-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-564sz\" (UID: \"67be1eda-e12d-4a81-9785-5c5c8c3989d2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:26.652082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.651908    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddpxt\" (UniqueName: \"kubernetes.io/projected/67be1eda-e12d-4a81-9785-5c5c8c3989d2-kube-api-access-ddpxt\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-564sz\" (UID: \"67be1eda-e12d-4a81-9785-5c5c8c3989d2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:26.652082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.651933    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67be1eda-e12d-4a81-9785-5c5c8c3989d2-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-564sz\" (UID: \"67be1eda-e12d-4a81-9785-5c5c8c3989d2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:26.652082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.651998    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/67be1eda-e12d-4a81-9785-5c5c8c3989d2-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-564sz\" (UID: \"67be1eda-e12d-4a81-9785-5c5c8c3989d2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:26.652729 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.652700    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67be1eda-e12d-4a81-9785-5c5c8c3989d2-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-564sz\" (UID: \"67be1eda-e12d-4a81-9785-5c5c8c3989d2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:26.652878 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.652858    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/67be1eda-e12d-4a81-9785-5c5c8c3989d2-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-564sz\" (UID: \"67be1eda-e12d-4a81-9785-5c5c8c3989d2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:26.654418 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.654398    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67be1eda-e12d-4a81-9785-5c5c8c3989d2-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-564sz\" (UID: \"67be1eda-e12d-4a81-9785-5c5c8c3989d2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:26.664986 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.664966    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddpxt\" (UniqueName: \"kubernetes.io/projected/67be1eda-e12d-4a81-9785-5c5c8c3989d2-kube-api-access-ddpxt\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-564sz\" (UID: \"67be1eda-e12d-4a81-9785-5c5c8c3989d2\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:26.696973 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.696897    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:26.758923 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:26.758886    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:27.031009 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:27.030985    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"]
Nov 03 19:43:27.033256 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:43:27.033225    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67be1eda_e12d_4a81_9785_5c5c8c3989d2.slice/crio-e5b6c04e413987a35cafcb543635a76b6a5a1255b9400cf44ac54dda486dd8b5 WatchSource:0}: Error finding container e5b6c04e413987a35cafcb543635a76b6a5a1255b9400cf44ac54dda486dd8b5: Status 404 returned error can't find the container with id e5b6c04e413987a35cafcb543635a76b6a5a1255b9400cf44ac54dda486dd8b5
Nov 03 19:43:27.624265 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:27.624229    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz" event={"ID":"67be1eda-e12d-4a81-9785-5c5c8c3989d2","Type":"ContainerStarted","Data":"e5b6c04e413987a35cafcb543635a76b6a5a1255b9400cf44ac54dda486dd8b5"}
Nov 03 19:43:34.644332 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:34.644300    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:35.930918 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:35.930877    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:36.007456 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.007422    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"]
Nov 03 19:43:36.012573 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.012548    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"
Nov 03 19:43:36.020310 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.020282    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"]
Nov 03 19:43:36.024104 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.023968    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-62j5q\""
Nov 03 19:43:36.024104 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.024072    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 19:43:36.072138 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.072106    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-76gw5"]
Nov 03 19:43:36.076088 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.076065    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-76gw5"
Nov 03 19:43:36.094148 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.094121    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-76gw5"]
Nov 03 19:43:36.102457 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.102433    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-4f7fr\""
Nov 03 19:43:36.103198 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.103172    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 19:43:36.103899 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.103824    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:36.145165 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.145130    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdlzz\" (UniqueName: \"kubernetes.io/projected/07f57b04-18f0-41ef-ab8c-e32fb889a586-kube-api-access-qdlzz\") pod \"llmisvc-controller-manager-869d5dcc5d-xbjmp\" (UID: \"07f57b04-18f0-41ef-ab8c-e32fb889a586\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"
Nov 03 19:43:36.145316 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.145187    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07f57b04-18f0-41ef-ab8c-e32fb889a586-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xbjmp\" (UID: \"07f57b04-18f0-41ef-ab8c-e32fb889a586\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"
Nov 03 19:43:36.246354 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.246316    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdlzz\" (UniqueName: \"kubernetes.io/projected/07f57b04-18f0-41ef-ab8c-e32fb889a586-kube-api-access-qdlzz\") pod \"llmisvc-controller-manager-869d5dcc5d-xbjmp\" (UID: \"07f57b04-18f0-41ef-ab8c-e32fb889a586\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"
Nov 03 19:43:36.246576 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.246367    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvhz4\" (UniqueName: \"kubernetes.io/projected/0f42d32f-c8f1-48b4-807a-76636151cb72-kube-api-access-pvhz4\") pod \"model-serving-api-75bd89fdfc-76gw5\" (UID: \"0f42d32f-c8f1-48b4-807a-76636151cb72\") " pod="opendatahub/model-serving-api-75bd89fdfc-76gw5"
Nov 03 19:43:36.246576 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.246396    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f42d32f-c8f1-48b4-807a-76636151cb72-tls-certs\") pod \"model-serving-api-75bd89fdfc-76gw5\" (UID: \"0f42d32f-c8f1-48b4-807a-76636151cb72\") " pod="opendatahub/model-serving-api-75bd89fdfc-76gw5"
Nov 03 19:43:36.246702 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.246595    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07f57b04-18f0-41ef-ab8c-e32fb889a586-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xbjmp\" (UID: \"07f57b04-18f0-41ef-ab8c-e32fb889a586\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"
Nov 03 19:43:36.249227 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.249203    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07f57b04-18f0-41ef-ab8c-e32fb889a586-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xbjmp\" (UID: \"07f57b04-18f0-41ef-ab8c-e32fb889a586\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"
Nov 03 19:43:36.273887 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.273861    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdlzz\" (UniqueName: \"kubernetes.io/projected/07f57b04-18f0-41ef-ab8c-e32fb889a586-kube-api-access-qdlzz\") pod \"llmisvc-controller-manager-869d5dcc5d-xbjmp\" (UID: \"07f57b04-18f0-41ef-ab8c-e32fb889a586\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"
Nov 03 19:43:36.310627 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.310592    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:36.329536 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.329506    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"
Nov 03 19:43:36.347901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.347872    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvhz4\" (UniqueName: \"kubernetes.io/projected/0f42d32f-c8f1-48b4-807a-76636151cb72-kube-api-access-pvhz4\") pod \"model-serving-api-75bd89fdfc-76gw5\" (UID: \"0f42d32f-c8f1-48b4-807a-76636151cb72\") " pod="opendatahub/model-serving-api-75bd89fdfc-76gw5"
Nov 03 19:43:36.348047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.347918    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f42d32f-c8f1-48b4-807a-76636151cb72-tls-certs\") pod \"model-serving-api-75bd89fdfc-76gw5\" (UID: \"0f42d32f-c8f1-48b4-807a-76636151cb72\") " pod="opendatahub/model-serving-api-75bd89fdfc-76gw5"
Nov 03 19:43:36.350737 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.350706    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f42d32f-c8f1-48b4-807a-76636151cb72-tls-certs\") pod \"model-serving-api-75bd89fdfc-76gw5\" (UID: \"0f42d32f-c8f1-48b4-807a-76636151cb72\") " pod="opendatahub/model-serving-api-75bd89fdfc-76gw5"
Nov 03 19:43:36.363230 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.363207    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvhz4\" (UniqueName: \"kubernetes.io/projected/0f42d32f-c8f1-48b4-807a-76636151cb72-kube-api-access-pvhz4\") pod \"model-serving-api-75bd89fdfc-76gw5\" (UID: \"0f42d32f-c8f1-48b4-807a-76636151cb72\") " pod="opendatahub/model-serving-api-75bd89fdfc-76gw5"
Nov 03 19:43:36.392910 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.392878    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-76gw5"
Nov 03 19:43:36.534417 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.534335    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:36.622246 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.622221    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"]
Nov 03 19:43:36.624602 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:43:36.624568    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod07f57b04_18f0_41ef_ab8c_e32fb889a586.slice/crio-8a3f142e1ac1e7d7f69a004cc7bbe079923952c2853004fd227ecaf73520366a WatchSource:0}: Error finding container 8a3f142e1ac1e7d7f69a004cc7bbe079923952c2853004fd227ecaf73520366a: Status 404 returned error can't find the container with id 8a3f142e1ac1e7d7f69a004cc7bbe079923952c2853004fd227ecaf73520366a
Nov 03 19:43:36.673372 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.673340    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp" event={"ID":"07f57b04-18f0-41ef-ab8c-e32fb889a586","Type":"ContainerStarted","Data":"8a3f142e1ac1e7d7f69a004cc7bbe079923952c2853004fd227ecaf73520366a"}
Nov 03 19:43:36.683931 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:36.683898    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-76gw5"]
Nov 03 19:43:36.686905 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:43:36.686874    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f42d32f_c8f1_48b4_807a_76636151cb72.slice/crio-4a7ef87c297eb7e077858a2a41a616296f5423f7a7fe6b27f657e39925e8e175 WatchSource:0}: Error finding container 4a7ef87c297eb7e077858a2a41a616296f5423f7a7fe6b27f657e39925e8e175: Status 404 returned error can't find the container with id 4a7ef87c297eb7e077858a2a41a616296f5423f7a7fe6b27f657e39925e8e175
Nov 03 19:43:37.607033 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:37.606903    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:37.682269 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:37.682228    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-76gw5" event={"ID":"0f42d32f-c8f1-48b4-807a-76636151cb72","Type":"ContainerStarted","Data":"4a7ef87c297eb7e077858a2a41a616296f5423f7a7fe6b27f657e39925e8e175"}
Nov 03 19:43:37.752876 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:37.752843    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:37.909094 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:37.909016    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:39.695812 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:39.695773    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp" event={"ID":"07f57b04-18f0-41ef-ab8c-e32fb889a586","Type":"ContainerStarted","Data":"c0cd1caf1cb6df251f06dfee8514c5cab78b86fc079c89bc9b7f34b619dab30e"}
Nov 03 19:43:39.696259 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:39.695934    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"
Nov 03 19:43:39.697471 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:39.697438    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-76gw5" event={"ID":"0f42d32f-c8f1-48b4-807a-76636151cb72","Type":"ContainerStarted","Data":"710321cbf9f14d9cf8fdd2c4f093e4d9c32c1174d3f72e4fda3321fcb4c51ef5"}
Nov 03 19:43:39.697619 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:39.697602    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-76gw5"
Nov 03 19:43:39.715026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:39.714964    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp" podStartSLOduration=2.272048493 podStartE2EDuration="4.714950494s" totalImagesPullingTime="2.442902001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:39.713553466 +0000 UTC m=+371.868455886" watchObservedRunningTime="2026-11-03 19:43:39.714950494 +0000 UTC m=+371.869852950"
Nov 03 19:43:39.733021 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:39.732969    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-76gw5" podStartSLOduration=1.535025073 podStartE2EDuration="3.732955642s" totalImagesPullingTime="2.197930569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:39.7318059 +0000 UTC m=+371.886708321" watchObservedRunningTime="2026-11-03 19:43:39.732955642 +0000 UTC m=+371.887858061"
Nov 03 19:43:42.715516 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:42.715455    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz" event={"ID":"67be1eda-e12d-4a81-9785-5c5c8c3989d2","Type":"ContainerStarted","Data":"a41e660c049356e899f07abb1ac159e130df90b32420d37ce82c7cbb032efd7a"}
Nov 03 19:43:42.715942 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:42.715601    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:42.737623 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:42.737571    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz" podStartSLOduration=1.340716806 podStartE2EDuration="16.737557093s" totalImagesPullingTime="15.396840287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:42.736700841 +0000 UTC m=+374.891603261" watchObservedRunningTime="2026-11-03 19:43:42.737557093 +0000 UTC m=+374.892459513"
Nov 03 19:43:44.319117 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:44.319082    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:45.215758 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:43:45.215723    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:43:50.707298 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:50.707263    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-76gw5"
Nov 03 19:43:53.726445 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:53.726415    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-564sz"
Nov 03 19:43:56.909651 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:56.909568    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:57.053674 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:57.053630    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:58.144185 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:58.144132    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:58.285033 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:58.284999    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:58.416657 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:58.416574    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:58.850766 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:58.850733    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:58.988064 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:43:58.988037    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:08.840162 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:08.840128    5659 generic.go:356] "Generic (PLEG): container finished" podID="9fa9e0c2-1bfa-4b41-b033-8d286b4c63be" containerID="08ed40a9b89e859e2e90ae0e800a75bf5769201a5fab5266bbb512c909c0bb4b" exitCode=0
Nov 03 19:44:08.840632 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:08.840202    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" event={"ID":"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be","Type":"ContainerDied","Data":"08ed40a9b89e859e2e90ae0e800a75bf5769201a5fab5266bbb512c909c0bb4b"}
Nov 03 19:44:08.840632 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:08.840598    5659 scope.go:122] "RemoveContainer" containerID="08ed40a9b89e859e2e90ae0e800a75bf5769201a5fab5266bbb512c909c0bb4b"
Nov 03 19:44:08.841701 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:08.841675    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:08.841777 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:08.841748    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:08.841814 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:08.841780    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:09.862540 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:09.862501    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" event={"ID":"9fa9e0c2-1bfa-4b41-b033-8d286b4c63be","Type":"ContainerStarted","Data":"e263b8dd5f936bab6fcea8661957ca7282c216b6ba268e73e5e3fe95dfb280fd"}
Nov 03 19:44:09.895535 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:09.895500    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:44:09.895704 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:09.895670    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnfsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Nov 03 19:44:09.895778 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:09.895721    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" podUID="9fa9e0c2-1bfa-4b41-b033-8d286b4c63be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Nov 03 19:44:10.431418 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.431385    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:10.627601 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.627566    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:10.705167 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.705141    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"
Nov 03 19:44:10.862723 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.862688    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"]
Nov 03 19:44:10.865727 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.865707    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:10.872505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.872473    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 19:44:10.875025 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.875003    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\""
Nov 03 19:44:10.875389 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.875370    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:44:10.875550 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.875471    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:44:10.877730 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.877709    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"]
Nov 03 19:44:10.896214 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.896186    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnfsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Nov 03 19:44:10.896346 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.896231    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" podUID="9fa9e0c2-1bfa-4b41-b033-8d286b4c63be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Nov 03 19:44:10.972230 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.972130    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-metrics-certs\") pod \"maas-api-7cd7866f98-ntwg2\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:10.972230 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.972187    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ffb51a7-07f2-4ae8-91da-250f92d7c155-openshift-service-ca\") pod \"maas-api-7cd7866f98-ntwg2\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:10.972475 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.972389    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr27q\" (UniqueName: \"kubernetes.io/projected/6ffb51a7-07f2-4ae8-91da-250f92d7c155-kube-api-access-hr27q\") pod \"maas-api-7cd7866f98-ntwg2\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:10.972475 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.972473    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-maas-api-tls\") pod \"maas-api-7cd7866f98-ntwg2\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:10.972724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:10.972700    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 19:44:11.073353 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:11.073311    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-metrics-certs\") pod \"maas-api-7cd7866f98-ntwg2\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:11.073353 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:11.073356    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ffb51a7-07f2-4ae8-91da-250f92d7c155-openshift-service-ca\") pod \"maas-api-7cd7866f98-ntwg2\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:11.073648 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:11.073405    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr27q\" (UniqueName: \"kubernetes.io/projected/6ffb51a7-07f2-4ae8-91da-250f92d7c155-kube-api-access-hr27q\") pod \"maas-api-7cd7866f98-ntwg2\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:11.073648 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:11.073432    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-maas-api-tls\") pod \"maas-api-7cd7866f98-ntwg2\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:11.074148 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:11.074119    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ffb51a7-07f2-4ae8-91da-250f92d7c155-openshift-service-ca\") pod \"maas-api-7cd7866f98-ntwg2\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:11.075901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:11.075878    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-metrics-certs\") pod \"maas-api-7cd7866f98-ntwg2\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:11.075995 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:11.075919    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-maas-api-tls\") pod \"maas-api-7cd7866f98-ntwg2\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:11.085660 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:11.085639    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr27q\" (UniqueName: \"kubernetes.io/projected/6ffb51a7-07f2-4ae8-91da-250f92d7c155-kube-api-access-hr27q\") pod \"maas-api-7cd7866f98-ntwg2\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") " pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:11.182374 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:11.182336    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:11.485436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:11.485412    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"]
Nov 03 19:44:11.487198 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:44:11.487170    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffb51a7_07f2_4ae8_91da_250f92d7c155.slice/crio-4ec07f2d9d2ed6ecf8cef9eb2285c85e5b0e08f28f2b1421f8b2ef10a385caaf WatchSource:0}: Error finding container 4ec07f2d9d2ed6ecf8cef9eb2285c85e5b0e08f28f2b1421f8b2ef10a385caaf: Status 404 returned error can't find the container with id 4ec07f2d9d2ed6ecf8cef9eb2285c85e5b0e08f28f2b1421f8b2ef10a385caaf
Nov 03 19:44:11.874441 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:11.874354    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2" event={"ID":"6ffb51a7-07f2-4ae8-91da-250f92d7c155","Type":"ContainerStarted","Data":"4ec07f2d9d2ed6ecf8cef9eb2285c85e5b0e08f28f2b1421f8b2ef10a385caaf"}
Nov 03 19:44:11.895951 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:11.895921    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnfsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Nov 03 19:44:11.896101 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:11.895971    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" podUID="9fa9e0c2-1bfa-4b41-b033-8d286b4c63be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Nov 03 19:44:12.896125 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:12.896092    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnfsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Nov 03 19:44:12.896741 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:12.896143    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" podUID="9fa9e0c2-1bfa-4b41-b033-8d286b4c63be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Nov 03 19:44:13.895879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:13.895842    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnfsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Nov 03 19:44:13.896069 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:13.895915    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" podUID="9fa9e0c2-1bfa-4b41-b033-8d286b4c63be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Nov 03 19:44:14.892183 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:14.892095    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2" event={"ID":"6ffb51a7-07f2-4ae8-91da-250f92d7c155","Type":"ContainerStarted","Data":"be081e1afbe0e73dc718dd0794486a41ce4c11dea535612addeac2b8e8893672"}
Nov 03 19:44:14.892183 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:14.892170    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:14.899663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:14.899631    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:44:14.899911 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:14.899887    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:44:14.900447 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:14.900432    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk"
Nov 03 19:44:14.912026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:14.911979    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2" podStartSLOduration=1.770374358 podStartE2EDuration="4.911964453s" totalImagesPullingTime="3.141590095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:14.911830197 +0000 UTC m=+407.066732617" watchObservedRunningTime="2026-11-03 19:44:14.911964453 +0000 UTC m=+407.066866873"
Nov 03 19:44:15.216087 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:44:15.216048    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:44:20.902198 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:20.902164    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 19:44:26.280583 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:26.280541    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:26.407793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:26.407759    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:34.087365 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:34.087324    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd"]
Nov 03 19:44:34.091020 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:34.090998    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd"
Nov 03 19:44:34.102074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:34.102050    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd"]
Nov 03 19:44:34.180087 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:34.180051    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgdwr\" (UniqueName: \"kubernetes.io/projected/950e6c44-5e74-4c6c-9c38-0712a81af85a-kube-api-access-tgdwr\") pod \"llmisvc-controller-manager-6466c554d8-qkjdd\" (UID: \"950e6c44-5e74-4c6c-9c38-0712a81af85a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd"
Nov 03 19:44:34.180087 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:34.180093    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/950e6c44-5e74-4c6c-9c38-0712a81af85a-cert\") pod \"llmisvc-controller-manager-6466c554d8-qkjdd\" (UID: \"950e6c44-5e74-4c6c-9c38-0712a81af85a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd"
Nov 03 19:44:34.280820 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:34.280767    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgdwr\" (UniqueName: \"kubernetes.io/projected/950e6c44-5e74-4c6c-9c38-0712a81af85a-kube-api-access-tgdwr\") pod \"llmisvc-controller-manager-6466c554d8-qkjdd\" (UID: \"950e6c44-5e74-4c6c-9c38-0712a81af85a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd"
Nov 03 19:44:34.280820 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:34.280833    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/950e6c44-5e74-4c6c-9c38-0712a81af85a-cert\") pod \"llmisvc-controller-manager-6466c554d8-qkjdd\" (UID: \"950e6c44-5e74-4c6c-9c38-0712a81af85a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd"
Nov 03 19:44:34.283299 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:34.283278    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/950e6c44-5e74-4c6c-9c38-0712a81af85a-cert\") pod \"llmisvc-controller-manager-6466c554d8-qkjdd\" (UID: \"950e6c44-5e74-4c6c-9c38-0712a81af85a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd"
Nov 03 19:44:34.293700 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:34.293675    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgdwr\" (UniqueName: \"kubernetes.io/projected/950e6c44-5e74-4c6c-9c38-0712a81af85a-kube-api-access-tgdwr\") pod \"llmisvc-controller-manager-6466c554d8-qkjdd\" (UID: \"950e6c44-5e74-4c6c-9c38-0712a81af85a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd"
Nov 03 19:44:34.404374 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:34.404283    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd"
Nov 03 19:44:34.667288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:34.667209    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:34.675843 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:34.675775    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd"]
Nov 03 19:44:34.677977 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:44:34.677948    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950e6c44_5e74_4c6c_9c38_0712a81af85a.slice/crio-258bf37ec6b912bebb02d61f93f5608489a620485050a32edc310094889b8ad3 WatchSource:0}: Error finding container 258bf37ec6b912bebb02d61f93f5608489a620485050a32edc310094889b8ad3: Status 404 returned error can't find the container with id 258bf37ec6b912bebb02d61f93f5608489a620485050a32edc310094889b8ad3
Nov 03 19:44:34.984751 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:34.984717    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd" event={"ID":"950e6c44-5e74-4c6c-9c38-0712a81af85a","Type":"ContainerStarted","Data":"258bf37ec6b912bebb02d61f93f5608489a620485050a32edc310094889b8ad3"}
Nov 03 19:44:35.991047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:35.991008    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd" event={"ID":"950e6c44-5e74-4c6c-9c38-0712a81af85a","Type":"ContainerStarted","Data":"9d2e1d0d11d84fe3929b981501287a035ee4ff4e139b0be14f4c612bc97fc583"}
Nov 03 19:44:35.991600 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:35.991120    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd"
Nov 03 19:44:36.010328 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:44:36.010269    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd" podStartSLOduration=1.641248212 podStartE2EDuration="2.010251305s" totalImagesPullingTime="369.003093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:36.010090647 +0000 UTC m=+428.164993067" watchObservedRunningTime="2026-11-03 19:44:36.010251305 +0000 UTC m=+428.165153725"
Nov 03 19:44:45.216139 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:44:45.216098    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:45:06.998649 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:06.998615    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qkjdd"
Nov 03 19:45:07.049664 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:07.049629    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"]
Nov 03 19:45:07.049912 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:07.049885    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp" podUID="07f57b04-18f0-41ef-ab8c-e32fb889a586" containerName="manager" containerID="cri-o://c0cd1caf1cb6df251f06dfee8514c5cab78b86fc079c89bc9b7f34b619dab30e" gracePeriod=30
Nov 03 19:45:07.295476 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:07.295447    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"
Nov 03 19:45:07.352160 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:07.352130    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f57b04-18f0-41ef-ab8c-e32fb889a586-kube-api-access-qdlzz\" (UniqueName: \"kubernetes.io/projected/07f57b04-18f0-41ef-ab8c-e32fb889a586-kube-api-access-qdlzz\") pod \"07f57b04-18f0-41ef-ab8c-e32fb889a586\" (UID: \"07f57b04-18f0-41ef-ab8c-e32fb889a586\") "
Nov 03 19:45:07.352317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:07.352172    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f57b04-18f0-41ef-ab8c-e32fb889a586-cert\" (UniqueName: \"kubernetes.io/secret/07f57b04-18f0-41ef-ab8c-e32fb889a586-cert\") pod \"07f57b04-18f0-41ef-ab8c-e32fb889a586\" (UID: \"07f57b04-18f0-41ef-ab8c-e32fb889a586\") "
Nov 03 19:45:07.354423 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:07.354394    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f57b04-18f0-41ef-ab8c-e32fb889a586-cert" pod "07f57b04-18f0-41ef-ab8c-e32fb889a586" (UID: "07f57b04-18f0-41ef-ab8c-e32fb889a586"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:45:07.354524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:07.354397    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f57b04-18f0-41ef-ab8c-e32fb889a586-kube-api-access-qdlzz" pod "07f57b04-18f0-41ef-ab8c-e32fb889a586" (UID: "07f57b04-18f0-41ef-ab8c-e32fb889a586"). InnerVolumeSpecName "kube-api-access-qdlzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:07.453430 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:07.453391    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdlzz\" (UniqueName: \"kubernetes.io/projected/07f57b04-18f0-41ef-ab8c-e32fb889a586-kube-api-access-qdlzz\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:07.453430 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:07.453423    5659 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07f57b04-18f0-41ef-ab8c-e32fb889a586-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:08.146795 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:08.146760    5659 generic.go:356] "Generic (PLEG): container finished" podID="07f57b04-18f0-41ef-ab8c-e32fb889a586" containerID="c0cd1caf1cb6df251f06dfee8514c5cab78b86fc079c89bc9b7f34b619dab30e" exitCode=0
Nov 03 19:45:08.147216 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:08.146810    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp" event={"ID":"07f57b04-18f0-41ef-ab8c-e32fb889a586","Type":"ContainerDied","Data":"c0cd1caf1cb6df251f06dfee8514c5cab78b86fc079c89bc9b7f34b619dab30e"}
Nov 03 19:45:08.147216 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:08.146815    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"
Nov 03 19:45:08.147216 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:08.146851    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp" event={"ID":"07f57b04-18f0-41ef-ab8c-e32fb889a586","Type":"ContainerDied","Data":"8a3f142e1ac1e7d7f69a004cc7bbe079923952c2853004fd227ecaf73520366a"}
Nov 03 19:45:08.147216 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:08.146871    5659 scope.go:122] "RemoveContainer" containerID="c0cd1caf1cb6df251f06dfee8514c5cab78b86fc079c89bc9b7f34b619dab30e"
Nov 03 19:45:08.155632 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:08.155615    5659 scope.go:122] "RemoveContainer" containerID="c0cd1caf1cb6df251f06dfee8514c5cab78b86fc079c89bc9b7f34b619dab30e"
Nov 03 19:45:08.155865 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:45:08.155847    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0cd1caf1cb6df251f06dfee8514c5cab78b86fc079c89bc9b7f34b619dab30e\": container with ID starting with c0cd1caf1cb6df251f06dfee8514c5cab78b86fc079c89bc9b7f34b619dab30e not found: ID does not exist" containerID="c0cd1caf1cb6df251f06dfee8514c5cab78b86fc079c89bc9b7f34b619dab30e"
Nov 03 19:45:08.155922 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:08.155871    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0cd1caf1cb6df251f06dfee8514c5cab78b86fc079c89bc9b7f34b619dab30e"} err="failed to get container status \"c0cd1caf1cb6df251f06dfee8514c5cab78b86fc079c89bc9b7f34b619dab30e\": rpc error: code = NotFound desc = could not find container \"c0cd1caf1cb6df251f06dfee8514c5cab78b86fc079c89bc9b7f34b619dab30e\": container with ID starting with c0cd1caf1cb6df251f06dfee8514c5cab78b86fc079c89bc9b7f34b619dab30e not found: ID does not exist"
Nov 03 19:45:08.163764 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:08.163746    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"]
Nov 03 19:45:08.169716 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:08.169697    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xbjmp"]
Nov 03 19:45:09.967696 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:09.967621    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f57b04-18f0-41ef-ab8c-e32fb889a586" path="/var/lib/kubelet/pods/07f57b04-18f0-41ef-ab8c-e32fb889a586/volumes"
Nov 03 19:45:15.216059 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:45:15.216017    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:45:23.616801 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:23.616771    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:23.774410 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:23.774381    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:26.121699 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:26.121661    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:28.215116 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:28.215080    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:28.344816 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:28.344734    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:34.716737 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:34.716701    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:45.215293 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:45:45.215256    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:45:54.112072 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:54.112037    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:54.615178 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:45:54.615140    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:05.828888 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:05.828853    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:05.969936 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:05.969902    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:09.131653 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:09.131616    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:09.362930 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:09.362900    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:09.487844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:09.487809    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:12.690684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:12.690636    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:13.280444 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:13.280409    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:14.541097 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:14.541063    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:15.220426 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:46:15.220392    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:46:18.593627 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:18.593590    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:20.145821 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:20.145787    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:20.570714 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:20.570666    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:34.774841 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:34.774803    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:36.369666 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:36.369633    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:36.750416 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:36.750354    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:36.877943 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:36.877908    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:38.221416 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:38.221382    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:38.348965 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:38.348926    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:45.216054 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:46:45.215980    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:46:54.011642 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.011603    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"]
Nov 03 19:46:54.012180 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.012153    5659 cpu_manager.go:414] "containerMap: removing container" podUID="07f57b04-18f0-41ef-ab8c-e32fb889a586" containerName="manager"
Nov 03 19:46:54.012180 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.012172    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f57b04-18f0-41ef-ab8c-e32fb889a586" containerName="manager"
Nov 03 19:46:54.012310 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.012268    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f57b04-18f0-41ef-ab8c-e32fb889a586" containerName="manager"
Nov 03 19:46:54.015505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.015460    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.028014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.027984    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"]
Nov 03 19:46:54.124555 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.124516    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.124555 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.124560    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.124765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.124653    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.124765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.124691    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.124765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.124751    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.124866 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.124780    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbqj9\" (UniqueName: \"kubernetes.io/projected/a16a0bd3-5dd9-496b-9708-ee0c4091090d-kube-api-access-nbqj9\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.124866 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.124811    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.124866 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.124834    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.124956 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.124896    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.225816 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.225768    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.225816 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.225818    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.226050 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.225916    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.226050 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.225940    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbqj9\" (UniqueName: \"kubernetes.io/projected/a16a0bd3-5dd9-496b-9708-ee0c4091090d-kube-api-access-nbqj9\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.226050 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.225968    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.226050 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.225990    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.226050 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.226017    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.226298 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.226082    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.226298 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.226111    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.226397 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.226372    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.226461 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.226416    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.226549 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.226476    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.226549 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.226505    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.226948 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.226930    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.228267 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.228246    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.228521 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.228501    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.238270 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.238238    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16a0bd3-5dd9-496b-9708-ee0c4091090d-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.238400 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.238386    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbqj9\" (UniqueName: \"kubernetes.io/projected/a16a0bd3-5dd9-496b-9708-ee0c4091090d-kube-api-access-nbqj9\") pod \"maas-default-gateway-openshift-default-855f9574cf-g44hx\" (UID: \"a16a0bd3-5dd9-496b-9708-ee0c4091090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.328630 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.328551    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:54.610181 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.610158    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"]
Nov 03 19:46:54.612372 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:46:54.612350    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16a0bd3_5dd9_496b_9708_ee0c4091090d.slice/crio-55060db9a68520b6fa230a075add12fcf202356dfe5a3d97959bd1f5bd1d8f55 WatchSource:0}: Error finding container 55060db9a68520b6fa230a075add12fcf202356dfe5a3d97959bd1f5bd1d8f55: Status 404 returned error can't find the container with id 55060db9a68520b6fa230a075add12fcf202356dfe5a3d97959bd1f5bd1d8f55
Nov 03 19:46:54.614459 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.614426    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:54.614573 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.614525    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:54.614633 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.614598    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:54.653775 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:54.653744    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx" event={"ID":"a16a0bd3-5dd9-496b-9708-ee0c4091090d","Type":"ContainerStarted","Data":"55060db9a68520b6fa230a075add12fcf202356dfe5a3d97959bd1f5bd1d8f55"}
Nov 03 19:46:55.662889 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:55.662855    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx" event={"ID":"a16a0bd3-5dd9-496b-9708-ee0c4091090d","Type":"ContainerStarted","Data":"bed8ed2ba87d8c33aefe5d3d9629efc7d1e94d25828999af52bad3c82d1f727d"}
Nov 03 19:46:55.690581 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:55.690523    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx" podStartSLOduration=2.6905110519999997 podStartE2EDuration="2.690511052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:55.689015919 +0000 UTC m=+567.843918346" watchObservedRunningTime="2026-11-03 19:46:55.690511052 +0000 UTC m=+567.845413471"
Nov 03 19:46:56.329537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:56.329504    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:46:56.330717 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:56.330693    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g44hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:46:56.330834 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:56.330739    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx" podUID="a16a0bd3-5dd9-496b-9708-ee0c4091090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 19:46:57.329708 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:57.329675    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g44hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:46:57.330103 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:57.329729    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx" podUID="a16a0bd3-5dd9-496b-9708-ee0c4091090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 19:46:58.328944 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:58.328896    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g44hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:46:58.329117 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:58.328978    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx" podUID="a16a0bd3-5dd9-496b-9708-ee0c4091090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 19:46:59.329196 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:59.329162    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g44hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:46:59.329604 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:46:59.329228    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx" podUID="a16a0bd3-5dd9-496b-9708-ee0c4091090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 19:47:00.329788 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:00.329747    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g44hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:47:00.330177 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:00.329810    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx" podUID="a16a0bd3-5dd9-496b-9708-ee0c4091090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 19:47:01.329956 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:01.329913    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g44hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:47:01.330371 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:01.329998    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx" podUID="a16a0bd3-5dd9-496b-9708-ee0c4091090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 19:47:02.848076 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:02.848036    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:47:02.848596 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:02.848517    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:47:02.849274 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:02.849255    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx"
Nov 03 19:47:04.324395 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:04.324359    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:04.468956 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:04.468926    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:04.608518 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:04.608428    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:04.818878 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:04.818837    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:04.986155 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:04.986111    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:05.124347 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:05.124314    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:05.314122 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:05.314033    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:05.445438 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:05.445404    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:05.633262 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:05.633181    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:06.256659 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:06.256619    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:06.688898 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:06.688808    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:06.819042 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:06.819010    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:07.337494 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:07.337454    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:07.469422 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:07.469388    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:08.540522 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:08.540474    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:08.672478 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:08.672433    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:09.240511 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:09.240451    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:09.373353 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:09.373319    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:09.511022 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:09.510946    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:09.653578 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:09.653549    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:09.791656 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:09.791562    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:09.943820 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:09.943779    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:10.081807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:10.081710    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:10.236831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:10.236788    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:10.391570 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:10.391494    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:10.710147 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:10.710116    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:11.218807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.218777    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:11.367578 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.367541    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:11.566928 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.566851    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"]
Nov 03 19:47:11.570889 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.570864    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:11.584349 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.584324    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"]
Nov 03 19:47:11.602491 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.602449    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nphs\" (UniqueName: \"kubernetes.io/projected/0068fc25-7f5d-4e0a-8941-7dd868693e27-kube-api-access-8nphs\") pod \"maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:11.602626 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.602503    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-maas-api-tls\") pod \"maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:11.602720 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.602635    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0068fc25-7f5d-4e0a-8941-7dd868693e27-openshift-service-ca\") pod \"maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:11.602781 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.602717    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-metrics-certs\") pod \"maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:11.605152 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.605138    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-745a88f9-serving-cert\""
Nov 03 19:47:11.659159 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.659128    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:11.704172 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.704134    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nphs\" (UniqueName: \"kubernetes.io/projected/0068fc25-7f5d-4e0a-8941-7dd868693e27-kube-api-access-8nphs\") pod \"maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:11.704349 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.704202    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-maas-api-tls\") pod \"maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:11.704349 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.704278    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0068fc25-7f5d-4e0a-8941-7dd868693e27-openshift-service-ca\") pod \"maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:11.704349 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.704321    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-metrics-certs\") pod \"maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:11.704565 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:47:11.704357    5659 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-serving-cert: secret "maas-api-e2e-ait-745a88f9-serving-cert" not found
Nov 03 19:47:11.704565 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:47:11.704429    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-maas-api-tls podName:0068fc25-7f5d-4e0a-8941-7dd868693e27 nodeName:}" failed. No retries permitted until 2026-11-03 19:47:12.204407093 +0000 UTC m=+584.359309499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-maas-api-tls") pod "maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq" (UID: "0068fc25-7f5d-4e0a-8941-7dd868693e27") : secret "maas-api-e2e-ait-745a88f9-serving-cert" not found
Nov 03 19:47:11.705177 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.705155    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0068fc25-7f5d-4e0a-8941-7dd868693e27-openshift-service-ca\") pod \"maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:11.706844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.706826    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-metrics-certs\") pod \"maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:11.717235 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.717217    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nphs\" (UniqueName: \"kubernetes.io/projected/0068fc25-7f5d-4e0a-8941-7dd868693e27-kube-api-access-8nphs\") pod \"maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:11.791249 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.791218    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:11.957454 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:11.957413    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:12.209704 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:12.209627    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-maas-api-tls\") pod \"maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:12.212146 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:12.212112    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-maas-api-tls\") pod \"maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:12.486081 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:12.485992    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:12.723862 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:12.723826    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:12.777748 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:12.777703    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"]
Nov 03 19:47:12.780966 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:47:12.780074    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0068fc25_7f5d_4e0a_8941_7dd868693e27.slice/crio-46cf4320c58888c786324ac098e72a16f0daebfd6ce0ef3646be93a08cf21014 WatchSource:0}: Error finding container 46cf4320c58888c786324ac098e72a16f0daebfd6ce0ef3646be93a08cf21014: Status 404 returned error can't find the container with id 46cf4320c58888c786324ac098e72a16f0daebfd6ce0ef3646be93a08cf21014
Nov 03 19:47:12.782080 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:12.782058    5659 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:47:12.851563 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:12.851527    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:13.079002 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:13.078912    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:13.208659 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:13.208620    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:13.404650 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:13.404573    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:13.553973 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:13.553937    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:13.761047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:13.761007    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq" event={"ID":"0068fc25-7f5d-4e0a-8941-7dd868693e27","Type":"ContainerStarted","Data":"50002976ddda5866bce8d26671bbdd3bfd13f70d018c681dcb133fe0ff656b58"}
Nov 03 19:47:13.761562 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:13.761074    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq" event={"ID":"0068fc25-7f5d-4e0a-8941-7dd868693e27","Type":"ContainerStarted","Data":"46cf4320c58888c786324ac098e72a16f0daebfd6ce0ef3646be93a08cf21014"}
Nov 03 19:47:13.761945 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:13.761923    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:13.794258 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:13.794188    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq" podStartSLOduration=2.342384944 podStartE2EDuration="2.79416682s" totalImagesPullingTime="451.781876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:13.79392204 +0000 UTC m=+585.948824458" watchObservedRunningTime="2026-11-03 19:47:13.79416682 +0000 UTC m=+585.949069240"
Nov 03 19:47:13.869550 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:13.869476    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:14.018503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.018392    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:14.388455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.388362    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"]
Nov 03 19:47:14.392204 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.392184    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.400547 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.400525    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3f7e10b8-serving-cert\""
Nov 03 19:47:14.402381 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.402360    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"]
Nov 03 19:47:14.431829 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.431797    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-maas-api-tls\") pod \"maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.431955 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.431896    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0766d6-9d54-407c-b86d-fbf0ef85822d-openshift-service-ca\") pod \"maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.432023 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.431959    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-metrics-certs\") pod \"maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.432023 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.431999    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lgx8\" (UniqueName: \"kubernetes.io/projected/0b0766d6-9d54-407c-b86d-fbf0ef85822d-kube-api-access-7lgx8\") pod \"maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.477283 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.477257    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:14.533386 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.533338    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-metrics-certs\") pod \"maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.533618 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.533403    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lgx8\" (UniqueName: \"kubernetes.io/projected/0b0766d6-9d54-407c-b86d-fbf0ef85822d-kube-api-access-7lgx8\") pod \"maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.533618 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.533529    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-maas-api-tls\") pod \"maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.533618 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.533614    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0766d6-9d54-407c-b86d-fbf0ef85822d-openshift-service-ca\") pod \"maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.534320 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.534296    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0766d6-9d54-407c-b86d-fbf0ef85822d-openshift-service-ca\") pod \"maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.535933 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.535913    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-maas-api-tls\") pod \"maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.536034 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.536018    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-metrics-certs\") pod \"maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.547052 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.547025    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lgx8\" (UniqueName: \"kubernetes.io/projected/0b0766d6-9d54-407c-b86d-fbf0ef85822d-kube-api-access-7lgx8\") pod \"maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.616432 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.616399    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:14.706974 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.706944    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:14.991291 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:14.991258    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"]
Nov 03 19:47:14.993733 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:47:14.993698    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0766d6_9d54_407c_b86d_fbf0ef85822d.slice/crio-6892cc1ca1ee5b390ed7b030fb268323b1e9d8341c4e1f486ada0e9f6879b1cf WatchSource:0}: Error finding container 6892cc1ca1ee5b390ed7b030fb268323b1e9d8341c4e1f486ada0e9f6879b1cf: Status 404 returned error can't find the container with id 6892cc1ca1ee5b390ed7b030fb268323b1e9d8341c4e1f486ada0e9f6879b1cf
Nov 03 19:47:15.216273 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:47:15.216234    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:47:15.305423 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:15.305324    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:15.776449 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:15.776415    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps" event={"ID":"0b0766d6-9d54-407c-b86d-fbf0ef85822d","Type":"ContainerStarted","Data":"3a16dca4ac56b5485c4a915052e93d3557eef2d90579efc5b8ac8e8bd5a60170"}
Nov 03 19:47:15.776654 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:15.776466    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps" event={"ID":"0b0766d6-9d54-407c-b86d-fbf0ef85822d","Type":"ContainerStarted","Data":"6892cc1ca1ee5b390ed7b030fb268323b1e9d8341c4e1f486ada0e9f6879b1cf"}
Nov 03 19:47:15.776654 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:15.776519    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:15.801246 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:15.801188    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps" podStartSLOduration=1.377995456 podStartE2EDuration="1.801173995s" totalImagesPullingTime="423.178539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:15.798517715 +0000 UTC m=+587.953420135" watchObservedRunningTime="2026-11-03 19:47:15.801173995 +0000 UTC m=+587.956076423"
Nov 03 19:47:16.547104 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:16.547056    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:16.684060 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:16.684020    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:17.367379 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:17.367288    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:17.781909 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:17.781867    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:17.929788 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:17.929757    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:18.161260 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:18.161184    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:19.097336 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:19.097295    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:19.976224 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:19.976193    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:20.689158 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.689126    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"]
Nov 03 19:47:20.692998 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.692974    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:20.698016 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.697992    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8c14f244-serving-cert\""
Nov 03 19:47:20.704859 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.704815    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"]
Nov 03 19:47:20.782759 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.782733    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:20.783743 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.783718    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:20.796259 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.796231    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38af2bf-f252-4e3d-85db-041eb1c0690e-openshift-service-ca\") pod \"maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:20.796379 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.796283    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-maas-api-tls\") pod \"maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:20.796379 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.796363    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4jqr\" (UniqueName: \"kubernetes.io/projected/f38af2bf-f252-4e3d-85db-041eb1c0690e-kube-api-access-x4jqr\") pod \"maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:20.796471 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.796415    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-metrics-certs\") pod \"maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:20.898295 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.898260    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38af2bf-f252-4e3d-85db-041eb1c0690e-openshift-service-ca\") pod \"maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:20.898584 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.898560    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-maas-api-tls\") pod \"maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:20.898731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.898715    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4jqr\" (UniqueName: \"kubernetes.io/projected/f38af2bf-f252-4e3d-85db-041eb1c0690e-kube-api-access-x4jqr\") pod \"maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:20.898844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.898829    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-metrics-certs\") pod \"maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:20.899050 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.899026    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38af2bf-f252-4e3d-85db-041eb1c0690e-openshift-service-ca\") pod \"maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:20.901567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.901543    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-metrics-certs\") pod \"maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:20.901723 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.901692    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-maas-api-tls\") pod \"maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:20.911504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.911454    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4jqr\" (UniqueName: \"kubernetes.io/projected/f38af2bf-f252-4e3d-85db-041eb1c0690e-kube-api-access-x4jqr\") pod \"maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:20.920341 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:20.920319    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:21.010545 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:21.010512    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:21.293752 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:21.293728    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"]
Nov 03 19:47:21.296339 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:47:21.296303    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38af2bf_f252_4e3d_85db_041eb1c0690e.slice/crio-6607e3e02cdba4a30b4f96b78dbf571c13b7b96f55093148de276aff5c399243 WatchSource:0}: Error finding container 6607e3e02cdba4a30b4f96b78dbf571c13b7b96f55093148de276aff5c399243: Status 404 returned error can't find the container with id 6607e3e02cdba4a30b4f96b78dbf571c13b7b96f55093148de276aff5c399243
Nov 03 19:47:21.752619 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:21.752590    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:21.788542 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:21.788506    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:21.815759 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:21.815718    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq" event={"ID":"f38af2bf-f252-4e3d-85db-041eb1c0690e","Type":"ContainerStarted","Data":"e1fa88ba828f70c4f0ef7e1db6d83b259f1f403140724c1fd5e94c7046969e90"}
Nov 03 19:47:21.815946 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:21.815815    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq" event={"ID":"f38af2bf-f252-4e3d-85db-041eb1c0690e","Type":"ContainerStarted","Data":"6607e3e02cdba4a30b4f96b78dbf571c13b7b96f55093148de276aff5c399243"}
Nov 03 19:47:21.816526 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:21.816467    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:22.619175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:22.619141    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:23.729765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:23.729732    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:24.559500 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:24.559450    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:25.408823 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:25.408792    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:26.301637 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:26.301602    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:27.056017 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:27.055985    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:27.876936 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:27.876907    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:28.830920 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:28.830887    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:28.842414 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:28.842391    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:28.851923 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:28.851881    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq" podStartSLOduration=8.488378297 podStartE2EDuration="8.851867408s" totalImagesPullingTime="363.489111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:21.838331675 +0000 UTC m=+593.993234096" watchObservedRunningTime="2026-11-03 19:47:28.851867408 +0000 UTC m=+601.006769826"
Nov 03 19:47:29.855620 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:29.855585    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:30.956156 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:30.956116    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:31.865027 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:31.864978    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:32.735562 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:32.735531    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:33.746285 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:33.746251    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:34.631809 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:34.631780    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:34.839918 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:34.839883    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:35.473733 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:35.473697    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:36.253805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:36.253774    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:37.108372 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:37.108339    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:38.028625 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:38.028592    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:38.369565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:38.369471    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:38.507594 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:38.507558    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:38.847551 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:38.847498    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:38.974035 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:38.974002    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:39.551088 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:39.551028    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:40.264173 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:40.264097    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:40.559817 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:40.559727    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:40.719338 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:40.719297    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:41.021304 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:41.021270    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:41.831194 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:41.831164    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:42.304526 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.304475    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"]
Nov 03 19:47:42.304827 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.304797    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq" podUID="0068fc25-7f5d-4e0a-8941-7dd868693e27" containerName="maas-api" containerID="cri-o://50002976ddda5866bce8d26671bbdd3bfd13f70d018c681dcb133fe0ff656b58" gracePeriod=30
Nov 03 19:47:42.384258 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.384230    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:42.516132 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.515697    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:42.595250 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.595213    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:42.661633 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.661457    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:42.711556 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.711524    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0068fc25-7f5d-4e0a-8941-7dd868693e27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0068fc25-7f5d-4e0a-8941-7dd868693e27-openshift-service-ca\") pod \"0068fc25-7f5d-4e0a-8941-7dd868693e27\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") "
Nov 03 19:47:42.711720 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.711576    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0068fc25-7f5d-4e0a-8941-7dd868693e27-kube-api-access-8nphs\" (UniqueName: \"kubernetes.io/projected/0068fc25-7f5d-4e0a-8941-7dd868693e27-kube-api-access-8nphs\") pod \"0068fc25-7f5d-4e0a-8941-7dd868693e27\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") "
Nov 03 19:47:42.711720 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.711635    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-maas-api-tls\") pod \"0068fc25-7f5d-4e0a-8941-7dd868693e27\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") "
Nov 03 19:47:42.711720 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.711712    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-metrics-certs\") pod \"0068fc25-7f5d-4e0a-8941-7dd868693e27\" (UID: \"0068fc25-7f5d-4e0a-8941-7dd868693e27\") "
Nov 03 19:47:42.711878 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.711854    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0068fc25-7f5d-4e0a-8941-7dd868693e27-openshift-service-ca" pod "0068fc25-7f5d-4e0a-8941-7dd868693e27" (UID: "0068fc25-7f5d-4e0a-8941-7dd868693e27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:42.712002 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.711987    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0068fc25-7f5d-4e0a-8941-7dd868693e27-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:42.713807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.713772    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-maas-api-tls" pod "0068fc25-7f5d-4e0a-8941-7dd868693e27" (UID: "0068fc25-7f5d-4e0a-8941-7dd868693e27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:42.713917 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.713813    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0068fc25-7f5d-4e0a-8941-7dd868693e27-kube-api-access-8nphs" pod "0068fc25-7f5d-4e0a-8941-7dd868693e27" (UID: "0068fc25-7f5d-4e0a-8941-7dd868693e27"). InnerVolumeSpecName "kube-api-access-8nphs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:42.713917 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.713856    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-metrics-certs" pod "0068fc25-7f5d-4e0a-8941-7dd868693e27" (UID: "0068fc25-7f5d-4e0a-8941-7dd868693e27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:42.795040 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.795009    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:42.813252 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.813217    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nphs\" (UniqueName: \"kubernetes.io/projected/0068fc25-7f5d-4e0a-8941-7dd868693e27-kube-api-access-8nphs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:42.813252 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.813247    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:42.813252 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.813257    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0068fc25-7f5d-4e0a-8941-7dd868693e27-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:42.923650 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.923558    5659 generic.go:356] "Generic (PLEG): container finished" podID="0068fc25-7f5d-4e0a-8941-7dd868693e27" containerID="50002976ddda5866bce8d26671bbdd3bfd13f70d018c681dcb133fe0ff656b58" exitCode=0
Nov 03 19:47:42.923650 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.923615    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"
Nov 03 19:47:42.923650 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.923636    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq" event={"ID":"0068fc25-7f5d-4e0a-8941-7dd868693e27","Type":"ContainerDied","Data":"50002976ddda5866bce8d26671bbdd3bfd13f70d018c681dcb133fe0ff656b58"}
Nov 03 19:47:42.924213 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.923689    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq" event={"ID":"0068fc25-7f5d-4e0a-8941-7dd868693e27","Type":"ContainerDied","Data":"46cf4320c58888c786324ac098e72a16f0daebfd6ce0ef3646be93a08cf21014"}
Nov 03 19:47:42.924213 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.923714    5659 scope.go:122] "RemoveContainer" containerID="50002976ddda5866bce8d26671bbdd3bfd13f70d018c681dcb133fe0ff656b58"
Nov 03 19:47:42.929911 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.929890    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:42.933438 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.933421    5659 scope.go:122] "RemoveContainer" containerID="50002976ddda5866bce8d26671bbdd3bfd13f70d018c681dcb133fe0ff656b58"
Nov 03 19:47:42.933752 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:47:42.933735    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50002976ddda5866bce8d26671bbdd3bfd13f70d018c681dcb133fe0ff656b58\": container with ID starting with 50002976ddda5866bce8d26671bbdd3bfd13f70d018c681dcb133fe0ff656b58 not found: ID does not exist" containerID="50002976ddda5866bce8d26671bbdd3bfd13f70d018c681dcb133fe0ff656b58"
Nov 03 19:47:42.933809 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.933761    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50002976ddda5866bce8d26671bbdd3bfd13f70d018c681dcb133fe0ff656b58"} err="failed to get container status \"50002976ddda5866bce8d26671bbdd3bfd13f70d018c681dcb133fe0ff656b58\": rpc error: code = NotFound desc = could not find container \"50002976ddda5866bce8d26671bbdd3bfd13f70d018c681dcb133fe0ff656b58\": container with ID starting with 50002976ddda5866bce8d26671bbdd3bfd13f70d018c681dcb133fe0ff656b58 not found: ID does not exist"
Nov 03 19:47:42.946847 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.946822    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"]
Nov 03 19:47:42.952475 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:42.952451    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-745a88f9-76945c9cd6-dmnwq"]
Nov 03 19:47:43.062381 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:43.062347    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:43.202732 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:43.202656    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:43.536663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:43.536633    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:43.732198 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:43.732161    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"]
Nov 03 19:47:43.732443 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:43.732414    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps" podUID="0b0766d6-9d54-407c-b86d-fbf0ef85822d" containerName="maas-api" containerID="cri-o://3a16dca4ac56b5485c4a915052e93d3557eef2d90579efc5b8ac8e8bd5a60170" gracePeriod=30
Nov 03 19:47:43.785303 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:43.785250    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:43.933154 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:43.932997    5659 generic.go:356] "Generic (PLEG): container finished" podID="0b0766d6-9d54-407c-b86d-fbf0ef85822d" containerID="3a16dca4ac56b5485c4a915052e93d3557eef2d90579efc5b8ac8e8bd5a60170" exitCode=0
Nov 03 19:47:43.933154 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:43.933069    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps" event={"ID":"0b0766d6-9d54-407c-b86d-fbf0ef85822d","Type":"ContainerDied","Data":"3a16dca4ac56b5485c4a915052e93d3557eef2d90579efc5b8ac8e8bd5a60170"}
Nov 03 19:47:43.935171 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:43.935108    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:43.969558 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:43.969524    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0068fc25-7f5d-4e0a-8941-7dd868693e27" path="/var/lib/kubelet/pods/0068fc25-7f5d-4e0a-8941-7dd868693e27/volumes"
Nov 03 19:47:44.009278 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.009257    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:44.079041 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.079005    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.126713 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.126619    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-metrics-certs\") pod \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") "
Nov 03 19:47:44.126891 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.126748    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-maas-api-tls\") pod \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") "
Nov 03 19:47:44.126891 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.126777    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0766d6-9d54-407c-b86d-fbf0ef85822d-kube-api-access-7lgx8\" (UniqueName: \"kubernetes.io/projected/0b0766d6-9d54-407c-b86d-fbf0ef85822d-kube-api-access-7lgx8\") pod \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") "
Nov 03 19:47:44.127007 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.126905    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0766d6-9d54-407c-b86d-fbf0ef85822d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0766d6-9d54-407c-b86d-fbf0ef85822d-openshift-service-ca\") pod \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\" (UID: \"0b0766d6-9d54-407c-b86d-fbf0ef85822d\") "
Nov 03 19:47:44.127436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.127294    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0766d6-9d54-407c-b86d-fbf0ef85822d-openshift-service-ca" pod "0b0766d6-9d54-407c-b86d-fbf0ef85822d" (UID: "0b0766d6-9d54-407c-b86d-fbf0ef85822d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:44.129033 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.129006    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0766d6-9d54-407c-b86d-fbf0ef85822d-kube-api-access-7lgx8" pod "0b0766d6-9d54-407c-b86d-fbf0ef85822d" (UID: "0b0766d6-9d54-407c-b86d-fbf0ef85822d"). InnerVolumeSpecName "kube-api-access-7lgx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:44.129033 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.129011    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-metrics-certs" pod "0b0766d6-9d54-407c-b86d-fbf0ef85822d" (UID: "0b0766d6-9d54-407c-b86d-fbf0ef85822d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:44.129255 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.129240    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-maas-api-tls" pod "0b0766d6-9d54-407c-b86d-fbf0ef85822d" (UID: "0b0766d6-9d54-407c-b86d-fbf0ef85822d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:44.209059 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.209026    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.227894 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.227860    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:44.227894 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.227888    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lgx8\" (UniqueName: \"kubernetes.io/projected/0b0766d6-9d54-407c-b86d-fbf0ef85822d-kube-api-access-7lgx8\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:44.227894 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.227899    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0766d6-9d54-407c-b86d-fbf0ef85822d-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:44.228191 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.227908    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0766d6-9d54-407c-b86d-fbf0ef85822d-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:44.351312 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.351279    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.494938 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.494902    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.622550 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.622512    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.882046 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.881967    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:44.940068 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.940041    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"
Nov 03 19:47:44.940068 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.940049    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps" event={"ID":"0b0766d6-9d54-407c-b86d-fbf0ef85822d","Type":"ContainerDied","Data":"6892cc1ca1ee5b390ed7b030fb268323b1e9d8341c4e1f486ada0e9f6879b1cf"}
Nov 03 19:47:44.940652 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.940113    5659 scope.go:122] "RemoveContainer" containerID="3a16dca4ac56b5485c4a915052e93d3557eef2d90579efc5b8ac8e8bd5a60170"
Nov 03 19:47:44.963041 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.963010    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"]
Nov 03 19:47:44.967299 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:44.967271    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f7e10b8-c7864cb48-lhgps"]
Nov 03 19:47:45.198791 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:45.198712    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:45.216075 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:47:45.216040    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:47:45.692791 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:45.692754    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:45.896244 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:45.896206    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:45.968088 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:45.968057    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0766d6-9d54-407c-b86d-fbf0ef85822d" path="/var/lib/kubelet/pods/0b0766d6-9d54-407c-b86d-fbf0ef85822d/volumes"
Nov 03 19:47:46.049227 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:46.049188    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:46.181627 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:46.181591    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:46.324880 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:46.324795    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:46.724050 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:46.724019    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:46.944943 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:46.944910    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:46.998396 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:46.998308    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"]
Nov 03 19:47:46.998857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:46.998561    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq" podUID="f38af2bf-f252-4e3d-85db-041eb1c0690e" containerName="maas-api" containerID="cri-o://e1fa88ba828f70c4f0ef7e1db6d83b259f1f403140724c1fd5e94c7046969e90" gracePeriod=30
Nov 03 19:47:47.098654 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.098619    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:47.300324 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.300300    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:47.301312 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.301290    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:47.353774 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.353734    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f38af2bf-f252-4e3d-85db-041eb1c0690e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38af2bf-f252-4e3d-85db-041eb1c0690e-openshift-service-ca\") pod \"f38af2bf-f252-4e3d-85db-041eb1c0690e\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") "
Nov 03 19:47:47.353968 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.353784    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-maas-api-tls\") pod \"f38af2bf-f252-4e3d-85db-041eb1c0690e\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") "
Nov 03 19:47:47.353968 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.353822    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38af2bf-f252-4e3d-85db-041eb1c0690e-kube-api-access-x4jqr\" (UniqueName: \"kubernetes.io/projected/f38af2bf-f252-4e3d-85db-041eb1c0690e-kube-api-access-x4jqr\") pod \"f38af2bf-f252-4e3d-85db-041eb1c0690e\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") "
Nov 03 19:47:47.353968 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.353920    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-metrics-certs\") pod \"f38af2bf-f252-4e3d-85db-041eb1c0690e\" (UID: \"f38af2bf-f252-4e3d-85db-041eb1c0690e\") "
Nov 03 19:47:47.354174 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.354154    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38af2bf-f252-4e3d-85db-041eb1c0690e-openshift-service-ca" pod "f38af2bf-f252-4e3d-85db-041eb1c0690e" (UID: "f38af2bf-f252-4e3d-85db-041eb1c0690e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:47.356349 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.356120    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-maas-api-tls" pod "f38af2bf-f252-4e3d-85db-041eb1c0690e" (UID: "f38af2bf-f252-4e3d-85db-041eb1c0690e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:47.356504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.356460    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38af2bf-f252-4e3d-85db-041eb1c0690e-kube-api-access-x4jqr" pod "f38af2bf-f252-4e3d-85db-041eb1c0690e" (UID: "f38af2bf-f252-4e3d-85db-041eb1c0690e"). InnerVolumeSpecName "kube-api-access-x4jqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:47.356563 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.356544    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-metrics-certs" pod "f38af2bf-f252-4e3d-85db-041eb1c0690e" (UID: "f38af2bf-f252-4e3d-85db-041eb1c0690e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:47.437317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.437287    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:47.454576 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.454540    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:47.454576 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.454567    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38af2bf-f252-4e3d-85db-041eb1c0690e-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:47.454576 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.454577    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f38af2bf-f252-4e3d-85db-041eb1c0690e-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:47.454576 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.454586    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4jqr\" (UniqueName: \"kubernetes.io/projected/f38af2bf-f252-4e3d-85db-041eb1c0690e-kube-api-access-x4jqr\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:47.612223 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.612140    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:47.758604 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.758568    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:47.887139 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.887058    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:47.957243 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.957207    5659 generic.go:356] "Generic (PLEG): container finished" podID="f38af2bf-f252-4e3d-85db-041eb1c0690e" containerID="e1fa88ba828f70c4f0ef7e1db6d83b259f1f403140724c1fd5e94c7046969e90" exitCode=0
Nov 03 19:47:47.957422 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.957252    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"
Nov 03 19:47:47.957422 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.957293    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq" event={"ID":"f38af2bf-f252-4e3d-85db-041eb1c0690e","Type":"ContainerDied","Data":"e1fa88ba828f70c4f0ef7e1db6d83b259f1f403140724c1fd5e94c7046969e90"}
Nov 03 19:47:47.957422 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.957348    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq" event={"ID":"f38af2bf-f252-4e3d-85db-041eb1c0690e","Type":"ContainerDied","Data":"6607e3e02cdba4a30b4f96b78dbf571c13b7b96f55093148de276aff5c399243"}
Nov 03 19:47:47.957422 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.957377    5659 scope.go:122] "RemoveContainer" containerID="e1fa88ba828f70c4f0ef7e1db6d83b259f1f403140724c1fd5e94c7046969e90"
Nov 03 19:47:47.967926 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.967908    5659 scope.go:122] "RemoveContainer" containerID="e1fa88ba828f70c4f0ef7e1db6d83b259f1f403140724c1fd5e94c7046969e90"
Nov 03 19:47:47.968278 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:47:47.968246    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fa88ba828f70c4f0ef7e1db6d83b259f1f403140724c1fd5e94c7046969e90\": container with ID starting with e1fa88ba828f70c4f0ef7e1db6d83b259f1f403140724c1fd5e94c7046969e90 not found: ID does not exist" containerID="e1fa88ba828f70c4f0ef7e1db6d83b259f1f403140724c1fd5e94c7046969e90"
Nov 03 19:47:47.968395 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.968290    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fa88ba828f70c4f0ef7e1db6d83b259f1f403140724c1fd5e94c7046969e90"} err="failed to get container status \"e1fa88ba828f70c4f0ef7e1db6d83b259f1f403140724c1fd5e94c7046969e90\": rpc error: code = NotFound desc = could not find container \"e1fa88ba828f70c4f0ef7e1db6d83b259f1f403140724c1fd5e94c7046969e90\": container with ID starting with e1fa88ba828f70c4f0ef7e1db6d83b259f1f403140724c1fd5e94c7046969e90 not found: ID does not exist"
Nov 03 19:47:47.984537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.983210    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"]
Nov 03 19:47:47.989239 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:47.989202    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8c14f244-587799c5b5-c6vmq"]
Nov 03 19:47:48.414239 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:48.414204    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:48.602351 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:48.602312    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:48.859968 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:48.859927    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:49.018109 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.018076    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:49.032126 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.032091    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"]
Nov 03 19:47:49.034725 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.034683    5659 cpu_manager.go:414] "containerMap: removing container" podUID="0b0766d6-9d54-407c-b86d-fbf0ef85822d" containerName="maas-api"
Nov 03 19:47:49.034725 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.034709    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0766d6-9d54-407c-b86d-fbf0ef85822d" containerName="maas-api"
Nov 03 19:47:49.034947 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.034752    5659 cpu_manager.go:414] "containerMap: removing container" podUID="0068fc25-7f5d-4e0a-8941-7dd868693e27" containerName="maas-api"
Nov 03 19:47:49.034947 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.034760    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0068fc25-7f5d-4e0a-8941-7dd868693e27" containerName="maas-api"
Nov 03 19:47:49.034947 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.034779    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f38af2bf-f252-4e3d-85db-041eb1c0690e" containerName="maas-api"
Nov 03 19:47:49.034947 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.034786    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38af2bf-f252-4e3d-85db-041eb1c0690e" containerName="maas-api"
Nov 03 19:47:49.034947 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.034884    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0766d6-9d54-407c-b86d-fbf0ef85822d" containerName="maas-api"
Nov 03 19:47:49.034947 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.034900    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="0068fc25-7f5d-4e0a-8941-7dd868693e27" containerName="maas-api"
Nov 03 19:47:49.034947 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.034909    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38af2bf-f252-4e3d-85db-041eb1c0690e" containerName="maas-api"
Nov 03 19:47:49.039743 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.039720    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.050461 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.050436    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3fbb1a-openshift-default-dockercfg-247ct\""
Nov 03 19:47:49.053387 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.053360    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"]
Nov 03 19:47:49.075896 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.075862    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.076047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.075915    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.076047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.076028    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.076190 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.076166    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.076288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.076264    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.076349 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.076327    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.076409 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.076395    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.076455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.076416    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6js\" (UniqueName: \"kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-kube-api-access-lx6js\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.076569 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.076548    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.160384 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.160290    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:49.177824 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.177791    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.177824 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.177829    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.178063 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.177930    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.178063 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.177956    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.178063 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.177974    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.178063 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.178021    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.178063 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.178058    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.178312 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.178118    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.178312 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.178250    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.178423 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.178310    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.178423 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.178363    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6js\" (UniqueName: \"kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-kube-api-access-lx6js\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.178423 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.178408    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.178724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.178364    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.179468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.179441    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.180804 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.180782    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.180931 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.180908    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.198194 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.198168    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.198403 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.198380    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6js\" (UniqueName: \"kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-kube-api-access-lx6js\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.307004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.306968    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:49.354056 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.354011    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:49.643193 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.643162    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"]
Nov 03 19:47:49.645467 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:47:49.645435    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0168a66b_a1c3_4bf2_a921_8cdff64b3ed4.slice/crio-fbcfea1ec973cd057bc7d3cd32141e5456273a3c7eaf430fafac4a300ceced94 WatchSource:0}: Error finding container fbcfea1ec973cd057bc7d3cd32141e5456273a3c7eaf430fafac4a300ceced94: Status 404 returned error can't find the container with id fbcfea1ec973cd057bc7d3cd32141e5456273a3c7eaf430fafac4a300ceced94
Nov 03 19:47:49.647986 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.647952    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:49.648100 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.648018    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:49.648100 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.648053    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:49.693981 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.693948    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:49.836743 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.836708    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:49.967737 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.967710    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f38af2bf-f252-4e3d-85db-041eb1c0690e" path="/var/lib/kubelet/pods/f38af2bf-f252-4e3d-85db-041eb1c0690e/volumes"
Nov 03 19:47:49.972411 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.972379    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" event={"ID":"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4","Type":"ContainerStarted","Data":"4efbec75d49d6c0d85b68fec79beabc7210dc1a5eeefa26cabb2c2442ea08a31"}
Nov 03 19:47:49.972550 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.972434    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" event={"ID":"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4","Type":"ContainerStarted","Data":"fbcfea1ec973cd057bc7d3cd32141e5456273a3c7eaf430fafac4a300ceced94"}
Nov 03 19:47:49.996317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:49.996035    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" podStartSLOduration=0.996016323 podStartE2EDuration="996.016323ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:49.995244244 +0000 UTC m=+622.150146677" watchObservedRunningTime="2026-11-03 19:47:49.996016323 +0000 UTC m=+622.150918741"
Nov 03 19:47:50.354364 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:50.354264    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:50.359785 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:50.359757    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:50.372389 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:50.372358    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:50.500681 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:50.500651    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:50.977813 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:50.977782    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:50.978766 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:50.978747    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:47:51.337652 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:51.337577    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:51.653201 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:51.653119    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:51.787900 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:51.787855    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:52.524347 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:52.524313    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:52.679979 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:52.679948    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:52.952705 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:52.952623    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:53.099346 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:53.099296    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:53.731065 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:53.731037    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:53.881892 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:53.881859    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:54.046421 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:54.046341    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:54.238566 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:54.238535    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:54.373782 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:54.373702    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:54.640778 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:54.640685    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:54.777213 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:54.777179    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:54.914760 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:54.914677    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:55.581880 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.581831    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"]
Nov 03 19:47:55.586315 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.586290    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:55.594645 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.591677    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-3fbb1a-serving-cert\""
Nov 03 19:47:55.594645 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.594577    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"]
Nov 03 19:47:55.638759 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.638727    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:55.638932 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.638802    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxv2\" (UniqueName: \"kubernetes.io/projected/2f904191-5cd6-4550-ad9d-431ba37472cd-kube-api-access-ssxv2\") pod \"maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:55.638932 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.638903    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-metrics-certs\") pod \"maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:55.639026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.638962    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f904191-5cd6-4550-ad9d-431ba37472cd-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:55.667149 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.667118    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:55.739897 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.739861    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxv2\" (UniqueName: \"kubernetes.io/projected/2f904191-5cd6-4550-ad9d-431ba37472cd-kube-api-access-ssxv2\") pod \"maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:55.739897 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.739899    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-metrics-certs\") pod \"maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:55.740141 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.739928    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f904191-5cd6-4550-ad9d-431ba37472cd-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:55.740141 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.739977    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:55.740787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.740757    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f904191-5cd6-4550-ad9d-431ba37472cd-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:55.742515 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.742477    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:55.742616 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.742539    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-metrics-certs\") pod \"maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:55.756440 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.756414    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxv2\" (UniqueName: \"kubernetes.io/projected/2f904191-5cd6-4550-ad9d-431ba37472cd-kube-api-access-ssxv2\") pod \"maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:55.803794 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.803769    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:55.902321 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:55.902231    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:56.208138 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:56.208110    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"]
Nov 03 19:47:56.210153 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:47:56.210126    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f904191_5cd6_4550_ad9d_431ba37472cd.slice/crio-0ee064ba02840d5100ff8042cf611209e3ce90a7a695144c319867393042c41f WatchSource:0}: Error finding container 0ee064ba02840d5100ff8042cf611209e3ce90a7a695144c319867393042c41f: Status 404 returned error can't find the container with id 0ee064ba02840d5100ff8042cf611209e3ce90a7a695144c319867393042c41f
Nov 03 19:47:56.771648 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:56.771589    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:57.010981 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:57.010938    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s" event={"ID":"2f904191-5cd6-4550-ad9d-431ba37472cd","Type":"ContainerStarted","Data":"8144380d26806e17efdf860f95a2ffe2fe016b660f58afd894b96f4bbed25b72"}
Nov 03 19:47:57.011354 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:57.010997    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s" event={"ID":"2f904191-5cd6-4550-ad9d-431ba37472cd","Type":"ContainerStarted","Data":"0ee064ba02840d5100ff8042cf611209e3ce90a7a695144c319867393042c41f"}
Nov 03 19:47:57.011354 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:57.011148    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:47:57.034707 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:57.034659    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s" podStartSLOduration=1.6388006050000001 podStartE2EDuration="2.034646705s" totalImagesPullingTime="395.8461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:57.032683997 +0000 UTC m=+629.187586419" watchObservedRunningTime="2026-11-03 19:47:57.034646705 +0000 UTC m=+629.189549123"
Nov 03 19:47:57.724160 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:57.724123    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:57.865591 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:57.865559    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:58.001685 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:58.001601    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:58.738144 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:58.738114    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:59.674111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:47:59.674079    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:00.664035 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:00.663998    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:01.499831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:01.499793    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:02.265161 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:02.265114    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:02.445506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:02.445092    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:03.022045 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:03.022015    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 19:48:03.146539 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:03.146507    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:03.283875 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:03.283797    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:04.055365 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:04.055324    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:04.888236 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:04.888194    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:05.768933 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:05.768896    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:05.901671 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:05.901609    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:06.754022 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:06.753993    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:06.886914 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:06.886871    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:07.609919 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:07.609881    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:07.735795 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:07.735756    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:08.289766 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:08.289730    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:08.437532 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:08.437500    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:08.858269 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:08.858180    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:09.924558 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:09.924529    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:10.085795 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:10.085709    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:10.111806 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:10.111776    5659 generic.go:356] "Generic (PLEG): container finished" podID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerID="4efbec75d49d6c0d85b68fec79beabc7210dc1a5eeefa26cabb2c2442ea08a31" exitCode=0
Nov 03 19:48:10.111979 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:10.111847    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" event={"ID":"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4","Type":"ContainerDied","Data":"4efbec75d49d6c0d85b68fec79beabc7210dc1a5eeefa26cabb2c2442ea08a31"}
Nov 03 19:48:10.112256 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:10.112231    5659 scope.go:122] "RemoveContainer" containerID="4efbec75d49d6c0d85b68fec79beabc7210dc1a5eeefa26cabb2c2442ea08a31"
Nov 03 19:48:10.113421 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:10.113395    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:48:10.113497 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:10.113461    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:48:10.113538 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:10.113509    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:48:10.681713 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:10.681675    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:10.825695 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:10.825667    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:11.069613 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:11.069578    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:11.118426 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:11.118392    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" event={"ID":"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4","Type":"ContainerStarted","Data":"c37184347e131df697cb225a8947a058c382cbf3e698a1c6ea07d302bdecffad"}
Nov 03 19:48:11.357561 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:11.356849    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:48:11.357561 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:11.357360    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 19:48:11.357561 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:11.357436    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 19:48:11.990717 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:11.990684    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:12.362560 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:12.361593    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 19:48:12.362560 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:12.361654    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 19:48:12.383755 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:12.383715    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:12.756300 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:12.756255    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:12.886347 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:12.886303    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:13.026243 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:13.026151    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:13.160767 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:13.160728    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:13.356563 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:13.355896    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 19:48:13.356563 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:13.355977    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 19:48:13.491269 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:13.491212    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:13.685502 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:13.685392    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:13.883365 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:13.883329    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:14.293214 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.293163    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"]
Nov 03 19:48:14.296723 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.296695    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.342508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.339757    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"openshift-service-ca.crt\""
Nov 03 19:48:14.342508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.340649    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"kube-root-ca.crt\""
Nov 03 19:48:14.342508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.340793    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"simulator-w2-2b5703-kserve-self-signed-certs\""
Nov 03 19:48:14.342508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.340882    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"default-dockercfg-4rsdq\""
Nov 03 19:48:14.342841 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.342690    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"]
Nov 03 19:48:14.354624 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.354597    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 19:48:14.355372 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.354874    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 19:48:14.384051 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.384027    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:14.468253 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.468216    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-dshm\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.468587 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.468556    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-home\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.468816 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.468797    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72080325-72f5-4713-a185-570a01746fc8-tls-certs\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.469041 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.469025    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zhpl\" (UniqueName: \"kubernetes.io/projected/72080325-72f5-4713-a185-570a01746fc8-kube-api-access-4zhpl\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.469171 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.469156    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-tmp-dir\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.469343 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.469330    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-model-cache\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.570970 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.570885    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72080325-72f5-4713-a185-570a01746fc8-tls-certs\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.570970 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.570932    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zhpl\" (UniqueName: \"kubernetes.io/projected/72080325-72f5-4713-a185-570a01746fc8-kube-api-access-4zhpl\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.571565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.570988    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-tmp-dir\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.571565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.571057    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-model-cache\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.571565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.571099    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-dshm\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.571565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.571173    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-home\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.571565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.571520    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-tmp-dir\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.571565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.571524    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-model-cache\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.571788 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.571592    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-home\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.573464 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.573441    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-dshm\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.574029 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.574009    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72080325-72f5-4713-a185-570a01746fc8-tls-certs\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.589056 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.589027    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zhpl\" (UniqueName: \"kubernetes.io/projected/72080325-72f5-4713-a185-570a01746fc8-kube-api-access-4zhpl\") pod \"simulator-w2-2b5703-kserve-55d9595866-ccrnw\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") " pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.607974 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.607947    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:14.775518 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.761950    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:14.918637 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:14.918614    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"]
Nov 03 19:48:14.920989 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:48:14.920816    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72080325_72f5_4713_a185_570a01746fc8.slice/crio-df5b56d9a52ba99dd96f61fe61a94d25f8c3a88efaf9fa2876e4e11eb7a32139 WatchSource:0}: Error finding container df5b56d9a52ba99dd96f61fe61a94d25f8c3a88efaf9fa2876e4e11eb7a32139: Status 404 returned error can't find the container with id df5b56d9a52ba99dd96f61fe61a94d25f8c3a88efaf9fa2876e4e11eb7a32139
Nov 03 19:48:15.148504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:15.148139    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" event={"ID":"72080325-72f5-4713-a185-570a01746fc8","Type":"ContainerStarted","Data":"df5b56d9a52ba99dd96f61fe61a94d25f8c3a88efaf9fa2876e4e11eb7a32139"}
Nov 03 19:48:15.218919 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:48:15.218866    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:48:15.368349 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:15.367893    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 03 19:48:15.368349 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:15.367954    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 03 19:48:15.456627 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:15.456520    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:16.163904 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.163846    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:16.258448 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.258411    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"]
Nov 03 19:48:16.262830 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.262778    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.271447 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.271413    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d3d0df\"/\"openshift-service-ca.crt\""
Nov 03 19:48:16.271611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.271573    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3d0df\"/\"default-dockercfg-rgjbk\""
Nov 03 19:48:16.271745 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.271726    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3d0df\"/\"simulator-w0-d3d0df-kserve-self-signed-certs\""
Nov 03 19:48:16.271840 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.271826    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d3d0df\"/\"kube-root-ca.crt\""
Nov 03 19:48:16.274390 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.274352    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"]
Nov 03 19:48:16.324234 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.324192    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:16.363554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.362082    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:48:16.363554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.363285    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:48:16.364968 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.364939    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 19:48:16.393777 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.393739    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-dshm\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.393972 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.393842    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7205a658-0441-4700-a96e-8323bf72b395-tls-certs\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.393972 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.393887    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-model-cache\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.394098 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.394048    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-home\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.394159 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.394109    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-tmp-dir\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.394215 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.394152    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rjbg\" (UniqueName: \"kubernetes.io/projected/7205a658-0441-4700-a96e-8323bf72b395-kube-api-access-6rjbg\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.494991 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.494951    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-dshm\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.495173 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.495019    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7205a658-0441-4700-a96e-8323bf72b395-tls-certs\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.495173 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.495048    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-model-cache\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.495173 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.495158    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-home\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.495346 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.495219    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-tmp-dir\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.495346 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.495260    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rjbg\" (UniqueName: \"kubernetes.io/projected/7205a658-0441-4700-a96e-8323bf72b395-kube-api-access-6rjbg\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.497380 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.497353    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-model-cache\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.497663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.497639    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-home\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.498157 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.498137    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-tmp-dir\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.498743 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.498720    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-dshm\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.500532 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.499883    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7205a658-0441-4700-a96e-8323bf72b395-tls-certs\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.513094 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.513067    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rjbg\" (UniqueName: \"kubernetes.io/projected/7205a658-0441-4700-a96e-8323bf72b395-kube-api-access-6rjbg\") pod \"simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") " pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.581916 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.581879    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:16.864180 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.864153    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"]
Nov 03 19:48:16.868025 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:48:16.867988    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7205a658_0441_4700_a96e_8323bf72b395.slice/crio-5654181d4790c89c83d245be9a572c3305f9c0cef1182d96d31f86af4beb9fdf WatchSource:0}: Error finding container 5654181d4790c89c83d245be9a572c3305f9c0cef1182d96d31f86af4beb9fdf: Status 404 returned error can't find the container with id 5654181d4790c89c83d245be9a572c3305f9c0cef1182d96d31f86af4beb9fdf
Nov 03 19:48:16.956863 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:16.956833    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:17.162399 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:17.162292    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" event={"ID":"7205a658-0441-4700-a96e-8323bf72b395","Type":"ContainerStarted","Data":"d8274317608e774f10d27cc611a9c4aec293fc5e7eb16ed7e3e0fe20c3832878"}
Nov 03 19:48:17.162399 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:17.162376    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" event={"ID":"7205a658-0441-4700-a96e-8323bf72b395","Type":"ContainerStarted","Data":"5654181d4790c89c83d245be9a572c3305f9c0cef1182d96d31f86af4beb9fdf"}
Nov 03 19:48:17.164205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:17.164176    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" event={"ID":"72080325-72f5-4713-a185-570a01746fc8","Type":"ContainerStarted","Data":"17ae96428fafb8554b2277c319828bfed909fcc7d2d6239aa6a6cee9dd3c51e4"}
Nov 03 19:48:17.181578 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:17.181531    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podStartSLOduration=1.1815170130000001 podStartE2EDuration="1.181517013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:17.180355036 +0000 UTC m=+649.335257456" watchObservedRunningTime="2026-11-03 19:48:17.181517013 +0000 UTC m=+649.336419426"
Nov 03 19:48:17.199431 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:17.199378    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podStartSLOduration=1.458679586 podStartE2EDuration="3.199364401s" totalImagesPullingTime="1.740684815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:17.198455883 +0000 UTC m=+649.353358308" watchObservedRunningTime="2026-11-03 19:48:17.199364401 +0000 UTC m=+649.354266820"
Nov 03 19:48:17.582741 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:17.582708    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:17.608410 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:17.608369    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:17.749627 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:17.749590    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:18.546579 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:18.546531    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:19.436426 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:19.436389    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:20.132555 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:20.132524    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:20.932633 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:20.932596    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:21.627218 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:21.627184    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:22.349147 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:22.349109    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:22.576866 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:22.576832    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:23.031319 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:23.031282    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:23.752087 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:23.752051    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:23.895875 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:23.895832    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:24.537872 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:24.537840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:24.608200 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:24.608169    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:24.621213 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:24.621179    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:25.219051 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:25.219019    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:48:25.327128 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:25.327095    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:26.039495 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:26.039446    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:26.582555 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:26.582512    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:26.595238 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:26.595208    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:26.740131 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:26.740095    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:27.234137 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:27.234098    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:48:27.641065 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:27.640952    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:27.942145 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:27.942060    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:28.464173 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:28.464137    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:29.150350 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:29.150320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:29.850170 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:29.850132    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:30.526288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:30.526255    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:31.339583 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:31.339539    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:32.231758 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:32.231722    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:32.931847 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:32.931807    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:33.645761 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:33.645723    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:34.332311 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:34.332278    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:34.894443 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:34.894405    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:35.039360 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.039320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:35.485497 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.485456    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"]
Nov 03 19:48:35.491110 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.491085    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.500603 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.500574    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3d0df\"/\"premium-w0-d3d0df-kserve-self-signed-certs\""
Nov 03 19:48:35.503144 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.503114    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"]
Nov 03 19:48:35.572505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.572445    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:35.595729 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.595683    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-model-cache\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.595961 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.595833    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-dshm\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.595961 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.595883    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c530754-24a3-4aa7-99cc-99bde27283b3-tls-certs\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.596076 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.596007    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfs6c\" (UniqueName: \"kubernetes.io/projected/0c530754-24a3-4aa7-99cc-99bde27283b3-kube-api-access-lfs6c\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.596130 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.596072    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-home\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.596252 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.596225    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-tmp-dir\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.697885 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.697776    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-dshm\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.697885 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.697839    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c530754-24a3-4aa7-99cc-99bde27283b3-tls-certs\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.697885 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.697890    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfs6c\" (UniqueName: \"kubernetes.io/projected/0c530754-24a3-4aa7-99cc-99bde27283b3-kube-api-access-lfs6c\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.698176 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.697940    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-home\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.698176 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.698007    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-tmp-dir\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.698176 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.698039    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-model-cache\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.698449 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.698422    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-model-cache\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.698586 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.698562    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-home\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.698785 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.698761    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-tmp-dir\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.700164 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.700144    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-dshm\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.700782 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.700764    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c530754-24a3-4aa7-99cc-99bde27283b3-tls-certs\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.710304 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.710275    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfs6c\" (UniqueName: \"kubernetes.io/projected/0c530754-24a3-4aa7-99cc-99bde27283b3-kube-api-access-lfs6c\") pod \"premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") " pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:35.751330 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.751246    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:35.804730 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:35.804687    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:36.084070 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:36.084040    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"]
Nov 03 19:48:36.085556 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:48:36.085532    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c530754_24a3_4aa7_99cc_99bde27283b3.slice/crio-c69cbbdb56294d176578ce112ec85f65d216aaf78a452ef07a5c47f09baaa9bf WatchSource:0}: Error finding container c69cbbdb56294d176578ce112ec85f65d216aaf78a452ef07a5c47f09baaa9bf: Status 404 returned error can't find the container with id c69cbbdb56294d176578ce112ec85f65d216aaf78a452ef07a5c47f09baaa9bf
Nov 03 19:48:36.276191 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:36.276151    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" event={"ID":"0c530754-24a3-4aa7-99cc-99bde27283b3","Type":"ContainerStarted","Data":"7aafe1e6315d850f3027f8ec212e633a96e2410bbefbfd3754202e3dab11587a"}
Nov 03 19:48:36.276191 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:36.276198    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" event={"ID":"0c530754-24a3-4aa7-99cc-99bde27283b3","Type":"ContainerStarted","Data":"c69cbbdb56294d176578ce112ec85f65d216aaf78a452ef07a5c47f09baaa9bf"}
Nov 03 19:48:36.296004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:36.295946    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podStartSLOduration=1.295929623 podStartE2EDuration="1.295929623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:36.295369561 +0000 UTC m=+668.450271981" watchObservedRunningTime="2026-11-03 19:48:36.295929623 +0000 UTC m=+668.450832060"
Nov 03 19:48:36.427812 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:36.427728    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:36.804951 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:36.804918    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:37.141369 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:37.141277    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:37.846397 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:37.846363    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:38.756149 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:38.756116    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:39.433068 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:39.433034    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:40.096656 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:40.096619    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:40.808141 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:40.808102    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:41.474815 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:41.474771    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:42.237893 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:42.237863    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:42.945333 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:42.945297    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:43.363275 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:43.363237    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:43.828775 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:43.828739    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:44.642113 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:44.642071    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:45.217558 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:48:45.217521    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:48:45.440529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:45.440497    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:45.805167 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:45.805127    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:45.818562 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:45.818525    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:46.130911 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.130836    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"]
Nov 03 19:48:46.134616 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.134591    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.138499 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.138459    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"kube-root-ca.crt\""
Nov 03 19:48:46.138663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.138502    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"premium-w4-3fbb1a-kserve-self-signed-certs\""
Nov 03 19:48:46.144431 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.144408    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"]
Nov 03 19:48:46.211227 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.211193    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:46.239477 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.239440    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"openshift-service-ca.crt\""
Nov 03 19:48:46.239477 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.239473    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"default-dockercfg-lxnx2\""
Nov 03 19:48:46.304640 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.304610    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tmp-dir\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.304810 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.304662    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-model-cache\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.304810 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.304749    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzqhm\" (UniqueName: \"kubernetes.io/projected/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-kube-api-access-xzqhm\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.304810 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.304803    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-home\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.304911 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.304894    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-dshm\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.304969 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.304951    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tls-certs\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.339099 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.339070    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:48:46.341378 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.341357    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:46.405828 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.405741    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tmp-dir\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.405828 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.405811    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-model-cache\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.406015 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.405838    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzqhm\" (UniqueName: \"kubernetes.io/projected/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-kube-api-access-xzqhm\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.406015 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.405876    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-home\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.406015 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.405905    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-dshm\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.406015 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.405946    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tls-certs\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.406245 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.406215    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tmp-dir\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.406339 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.406224    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-model-cache\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.406339 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.406267    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-home\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.408561 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.408478    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-dshm\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.408891 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.408860    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tls-certs\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.422315 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.422291    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzqhm\" (UniqueName: \"kubernetes.io/projected/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-kube-api-access-xzqhm\") pod \"premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") " pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.451131 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.451083    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:46.733376 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:46.733347    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"]
Nov 03 19:48:46.735249 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:48:46.735221    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb309ad2b_6391_4e09_8bbc_ae2ca1bdef85.slice/crio-9ddf2288a47411900936d1b58fdbeff56ef574bc35fb55bea75fd6258089c4ba WatchSource:0}: Error finding container 9ddf2288a47411900936d1b58fdbeff56ef574bc35fb55bea75fd6258089c4ba: Status 404 returned error can't find the container with id 9ddf2288a47411900936d1b58fdbeff56ef574bc35fb55bea75fd6258089c4ba
Nov 03 19:48:47.146756 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:47.146671    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:47.334477 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:47.334439    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" event={"ID":"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85","Type":"ContainerStarted","Data":"1466b9e49f1926675a97766a0ee53a5424b93d341d38bbb1383c74fbe34dc2be"}
Nov 03 19:48:47.334477 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:47.334495    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" event={"ID":"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85","Type":"ContainerStarted","Data":"9ddf2288a47411900936d1b58fdbeff56ef574bc35fb55bea75fd6258089c4ba"}
Nov 03 19:48:47.354590 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:47.354542    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podStartSLOduration=1.354528996 podStartE2EDuration="1.354528996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:47.353147653 +0000 UTC m=+679.508050072" watchObservedRunningTime="2026-11-03 19:48:47.354528996 +0000 UTC m=+679.509431414"
Nov 03 19:48:47.451666 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:47.451581    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:47.921043 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:47.921008    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:48.641817 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:48.641783    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:49.335436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:49.335399    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:50.034059 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:50.034028    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:51.062927 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:51.062887    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:52.121952 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:52.121887    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:52.869819 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:52.869790    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:53.146860 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.146764    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"]
Nov 03 19:48:53.151649 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.151622    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.160925 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.160768    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"premium-w1-32e482-kserve-self-signed-certs\""
Nov 03 19:48:53.160925 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.160873    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"kube-root-ca.crt\""
Nov 03 19:48:53.160925 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.160773    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"openshift-service-ca.crt\""
Nov 03 19:48:53.161519 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.161471    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"default-dockercfg-5r8nn\""
Nov 03 19:48:53.164894 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.164863    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"]
Nov 03 19:48:53.175609 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.175580    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-model-cache\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.175741 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.175666    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-dshm\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.175741 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.175699    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tls-certs\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.175870 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.175746    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tmp-dir\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.175870 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.175811    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-home\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.175870 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.175849    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzdl8\" (UniqueName: \"kubernetes.io/projected/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-kube-api-access-bzdl8\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.231238 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.231205    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:53.276707 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.276666    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-model-cache\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.276917 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.276739    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-dshm\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.276917 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.276767    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tls-certs\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.276917 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.276803    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tmp-dir\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.276917 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.276861    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-home\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.276917 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.276893    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzdl8\" (UniqueName: \"kubernetes.io/projected/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-kube-api-access-bzdl8\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.277250 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.277217    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-model-cache\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.277535 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.277510    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-home\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.277877 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.277853    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tmp-dir\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.279748 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.279727    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tls-certs\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.280307 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.280286    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-dshm\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.290317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.290292    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzdl8\" (UniqueName: \"kubernetes.io/projected/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-kube-api-access-bzdl8\") pod \"premium-w1-32e482-kserve-dd44fcc8c-jrcfw\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") " pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.484668 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.484630    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:53.746093 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.746009    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:53.762529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:53.762500    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"]
Nov 03 19:48:53.764246 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:48:53.764218    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fcdaebb_cfd8_451c_aae7_fd3603ffdfa9.slice/crio-40d91fef7f4e493ef204d85865d192e8044938d4567cdc834a314fcfb8cb1262 WatchSource:0}: Error finding container 40d91fef7f4e493ef204d85865d192e8044938d4567cdc834a314fcfb8cb1262: Status 404 returned error can't find the container with id 40d91fef7f4e493ef204d85865d192e8044938d4567cdc834a314fcfb8cb1262
Nov 03 19:48:54.386973 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:54.386939    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" event={"ID":"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9","Type":"ContainerStarted","Data":"0ef467b5b183c95c0eb03b83b3425b091c627eb44f27af52a12462fb07499ef1"}
Nov 03 19:48:54.386973 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:54.386980    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" event={"ID":"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9","Type":"ContainerStarted","Data":"40d91fef7f4e493ef204d85865d192e8044938d4567cdc834a314fcfb8cb1262"}
Nov 03 19:48:54.407808 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:54.407758    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podStartSLOduration=1.407742826 podStartE2EDuration="1.407742826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:54.406269023 +0000 UTC m=+686.561171448" watchObservedRunningTime="2026-11-03 19:48:54.407742826 +0000 UTC m=+686.562645245"
Nov 03 19:48:54.485584 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:54.485540    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:48:54.640571 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:54.640471    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:55.488206 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:55.488164    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:56.230503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:56.230442    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:56.451227 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:56.451196    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:56.465026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:56.464990    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:57.192778 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:57.192744    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:57.416007 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:57.415976    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:48:58.032956 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:58.032923    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:58.757084 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:58.757054    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:59.623960 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:48:59.623924    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:00.241422 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:00.241391    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:01.169733 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:01.169700    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:02.246578 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:02.246539    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:02.938490 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:02.938437    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:03.485549 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:03.485513    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:49:03.498326 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:03.498300    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:49:03.649599 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:03.649566    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:04.349068 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:04.349039    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:04.451331 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:04.451300    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:49:05.238160 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:05.238126    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:06.043657 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:06.043619    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:06.738696 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:06.738657    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:07.549793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:07.549758    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:08.353142 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:08.353110    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:09.136780 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:09.136740    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:09.936083 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:09.936042    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:10.761586 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:10.761553    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:11.557551 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:11.557501    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:12.410149 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:12.410115    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:13.341917 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:13.341881    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:14.239061 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:14.239004    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:14.940738 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:14.940697    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:15.216952 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:49:15.216921    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:49:15.731123 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:15.731078    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:16.457217 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:16.457183    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:17.147854 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:17.147821    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:17.858659 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:17.858626    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:18.649861 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:18.649830    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:19.605145 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:19.605106    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:20.453962 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:20.453929    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:21.263457 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:21.263427    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:22.144619 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:22.144581    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:22.945708 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:22.945679    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:23.769633 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:23.769594    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:24.476793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:24.476762    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:25.155686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:25.155656    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:25.834323 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:25.834287    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:26.590966 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:26.590930    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:27.357934 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:27.357899    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:28.140822 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:28.140787    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:28.850395 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:28.850363    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:29.639256 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:29.639224    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:30.520661 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.520607    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:30.586526 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.586471    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"]
Nov 03 19:49:30.592346 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.592320    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"
Nov 03 19:49:30.596887 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.596373    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:49:30.597348 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.597325    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\""
Nov 03 19:49:30.599320 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.599288    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"]
Nov 03 19:49:30.659879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.659833    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flmgd\" (UniqueName: \"kubernetes.io/projected/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-kube-api-access-flmgd\") pod \"maas-api-revoke-sub-ebd9ceca-m98f4\" (UID: \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"
Nov 03 19:49:30.660068 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.660020    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd9ceca-m98f4\" (UID: \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"
Nov 03 19:49:30.660123 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.660073    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd9ceca-m98f4\" (UID: \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"
Nov 03 19:49:30.761808 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.761758    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd9ceca-m98f4\" (UID: \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"
Nov 03 19:49:30.762016 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.761850    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd9ceca-m98f4\" (UID: \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"
Nov 03 19:49:30.762016 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.761887    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flmgd\" (UniqueName: \"kubernetes.io/projected/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-kube-api-access-flmgd\") pod \"maas-api-revoke-sub-ebd9ceca-m98f4\" (UID: \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"
Nov 03 19:49:30.762478 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.762452    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd9ceca-m98f4\" (UID: \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"
Nov 03 19:49:30.764167 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.764144    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd9ceca-m98f4\" (UID: \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"
Nov 03 19:49:30.774589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.774527    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flmgd\" (UniqueName: \"kubernetes.io/projected/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-kube-api-access-flmgd\") pod \"maas-api-revoke-sub-ebd9ceca-m98f4\" (UID: \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"
Nov 03 19:49:30.908654 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:30.908619    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"
Nov 03 19:49:31.192341 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:31.192316    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"]
Nov 03 19:49:31.194553 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:49:31.194525    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5251ea_f47a_4b1c_88ce_39a14a4cc8bf.slice/crio-5d8e43d2c085e6be0dd3292d28fcd5e19a0f2c438aaa0186dc35469b75874f77 WatchSource:0}: Error finding container 5d8e43d2c085e6be0dd3292d28fcd5e19a0f2c438aaa0186dc35469b75874f77: Status 404 returned error can't find the container with id 5d8e43d2c085e6be0dd3292d28fcd5e19a0f2c438aaa0186dc35469b75874f77
Nov 03 19:49:31.344078 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:31.344036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:31.599094 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:31.599053    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4" event={"ID":"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf","Type":"ContainerStarted","Data":"5d8e43d2c085e6be0dd3292d28fcd5e19a0f2c438aaa0186dc35469b75874f77"}
Nov 03 19:49:32.139862 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:32.139829    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:32.968843 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:32.968813    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:33.643931 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.643899    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:33.950385 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.950304    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"]
Nov 03 19:49:33.955817 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.955794    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:33.960141 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.960116    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-32e482-openshift-default-dockercfg-5pv2v\""
Nov 03 19:49:33.973222 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.973194    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"]
Nov 03 19:49:33.995868 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.995835    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:33.996040 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.995907    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:33.996188 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.996168    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:33.996266 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.996248    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:33.996364 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.996346    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfw4f\" (UniqueName: \"kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-kube-api-access-cfw4f\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:33.996408 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.996379    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:33.996408 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.996401    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:33.996478 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.996436    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:33.996595 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:33.996570    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.097550 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.097509    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfw4f\" (UniqueName: \"kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-kube-api-access-cfw4f\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.097758 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.097558    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.097758 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.097574    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.097758 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.097697    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.097944 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.097758    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.097944 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.097829    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.097944 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.097876    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.098123 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.097954    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.098123 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.097998    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.098123 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.098018    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.098123 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.098063    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.098289 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.098252    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.098335 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.098310    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.098424 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.098396    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.100011 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.099988    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.100432 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.100409    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.109826 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.109807    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.109978 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.109959    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfw4f\" (UniqueName: \"kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-kube-api-access-cfw4f\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.274301 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.274246    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:34.437129 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.437093    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:34.555844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.555820    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"]
Nov 03 19:49:34.558720 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:49:34.558695    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a5c8dc_4dfc_4108_bc92_88bc2ed23cad.slice/crio-58ddb86b5aa219672692b4d5d5674caf0d702892e5ffb48ea5e2d661ee3d8c96 WatchSource:0}: Error finding container 58ddb86b5aa219672692b4d5d5674caf0d702892e5ffb48ea5e2d661ee3d8c96: Status 404 returned error can't find the container with id 58ddb86b5aa219672692b4d5d5674caf0d702892e5ffb48ea5e2d661ee3d8c96
Nov 03 19:49:34.561069 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.561037    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:34.561149 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.561102    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:34.561149 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.561134    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:34.621228 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.621197    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp" event={"ID":"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad","Type":"ContainerStarted","Data":"58ddb86b5aa219672692b4d5d5674caf0d702892e5ffb48ea5e2d661ee3d8c96"}
Nov 03 19:49:34.949622 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:34.949522    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:35.271542 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:35.268328    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:35.629406 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:35.629314    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp" event={"ID":"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad","Type":"ContainerStarted","Data":"097e7cd7adc5884e46c4cadcb2ac1679fe3d07a55946e3686b643b95adace303"}
Nov 03 19:49:35.664852 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:35.664774    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp" podStartSLOduration=2.664754537 podStartE2EDuration="2.664754537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:35.663278899 +0000 UTC m=+727.818181313" watchObservedRunningTime="2026-11-03 19:49:35.664754537 +0000 UTC m=+727.819656957"
Nov 03 19:49:36.015064 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:36.015025    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:36.274535 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:36.274429    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:36.276160 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:36.276121    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 19:49:36.276318 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:36.276178    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp" podUID="26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 19:49:36.720066 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:36.720036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:37.274714 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:37.274679    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 19:49:37.275130 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:37.274743    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp" podUID="26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 19:49:37.427630 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:37.427564    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:38.136170 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:38.136073    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:38.274663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:38.274624    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 19:49:38.274864 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:38.274684    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp" podUID="26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 19:49:38.939107 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:38.939023    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:39.275114 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:39.275079    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 19:49:39.275540 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:39.275139    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp" podUID="26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 19:49:39.656565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:39.652674    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:40.275560 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:40.275513    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 19:49:40.276073 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:40.275577    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp" podUID="26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 19:49:40.350099 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:40.349949    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:41.051394 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:41.051359    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:41.742918 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:41.742873    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:41.863247 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:41.863215    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:41.863638 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:41.863613    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:41.864414 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:41.864397    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 19:49:42.500669 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:42.500636    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:43.149778 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:43.149737    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:43.855584 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:43.855548    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:44.542723 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:44.542685    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:44.708618 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:44.708583    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4" event={"ID":"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf","Type":"ContainerStarted","Data":"caff2f12e1d5ae91e0e0368002501fde20510066f8a575c0302f4893e2bf9a29"}
Nov 03 19:49:44.729365 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:44.729316    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4" podStartSLOduration=1.852286663 podStartE2EDuration="14.729299797s" totalImagesPullingTime="12.877013134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:44.726984174 +0000 UTC m=+736.881886593" watchObservedRunningTime="2026-11-03 19:49:44.729299797 +0000 UTC m=+736.884202215"
Nov 03 19:49:45.217464 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:49:45.217427    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:49:45.238185 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:45.238159    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:45.715656 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:45.715618    5659 generic.go:356] "Generic (PLEG): container finished" podID="bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf" containerID="caff2f12e1d5ae91e0e0368002501fde20510066f8a575c0302f4893e2bf9a29" exitCode=0
Nov 03 19:49:45.716156 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:45.715665    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4" event={"ID":"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf","Type":"ContainerDied","Data":"caff2f12e1d5ae91e0e0368002501fde20510066f8a575c0302f4893e2bf9a29"}
Nov 03 19:49:45.802732 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:45.802679    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:45.937223 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:45.937188    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:46.232842 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:46.232805    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:46.371430 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:46.371395    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:46.881883 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:46.881858    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"
Nov 03 19:49:46.960874 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:46.960838    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-kube-api-access-flmgd\" (UniqueName: \"kubernetes.io/projected/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-kube-api-access-flmgd\") pod \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\" (UID: \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\") "
Nov 03 19:49:46.961061 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:46.960948    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-openshift-ca\") pod \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\" (UID: \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\") "
Nov 03 19:49:46.961061 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:46.961043    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-xks-ca\") pod \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\" (UID: \"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf\") "
Nov 03 19:49:46.961425 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:46.961394    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-openshift-ca" pod "bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf" (UID: "bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:49:46.963311 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:46.963277    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-kube-api-access-flmgd" pod "bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf" (UID: "bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf"). InnerVolumeSpecName "kube-api-access-flmgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:46.963436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:46.963344    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-xks-ca" pod "bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf" (UID: "bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:49:47.063088 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:47.062993    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:47.063313 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:47.063286    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:47.063313 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:47.063313    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flmgd\" (UniqueName: \"kubernetes.io/projected/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf-kube-api-access-flmgd\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:47.153552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:47.153476    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:47.728613 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:47.728584    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"
Nov 03 19:49:47.728814 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:47.728624    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4" event={"ID":"bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf","Type":"ContainerDied","Data":"5d8e43d2c085e6be0dd3292d28fcd5e19a0f2c438aaa0186dc35469b75874f77"}
Nov 03 19:49:47.728814 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:47.728681    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d8e43d2c085e6be0dd3292d28fcd5e19a0f2c438aaa0186dc35469b75874f77"
Nov 03 19:49:48.080176 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:48.080081    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:48.889442 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:48.889403    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g44hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:49:48.889686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:48.889537    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx" podUID="a16a0bd3-5dd9-496b-9708-ee0c4091090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:48.942040 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:48.941990    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:49.755106 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:49.755073    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:50.562779 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:50.562748    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:51.471097 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:51.471066    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:52.474617 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:52.474583    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:53.170838 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:53.170807    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:53.971741 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:53.971711    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:54.781759 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:54.781723    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:55.550516 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:55.548637    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:56.304261 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:56.304228    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:57.137697 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:57.137661    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:58.059374 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.059338    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:58.376637 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.376539    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"]
Nov 03 19:49:58.377179 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.377153    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf" containerName="revoke-keys"
Nov 03 19:49:58.377357 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.377334    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf" containerName="revoke-keys"
Nov 03 19:49:58.377651 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.377499    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf" containerName="revoke-keys"
Nov 03 19:49:58.382704 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.382681    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.390699 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.390666    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"]
Nov 03 19:49:58.419306 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.419253    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"e2e-conflict-a-a0f38071-kserve-self-signed-certs\""
Nov 03 19:49:58.484510 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.484461    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:58.523621 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.523580    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-dshm\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.523867 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.523849    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35875954-d3a0-4289-bb78-76df37572027-tls-certs\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.529504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.524004    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlspx\" (UniqueName: \"kubernetes.io/projected/35875954-d3a0-4289-bb78-76df37572027-kube-api-access-jlspx\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.529736 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.529712    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-tmp-dir\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.529872 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.529855    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-model-cache\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.530074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.530057    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-home\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.631578 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.631445    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlspx\" (UniqueName: \"kubernetes.io/projected/35875954-d3a0-4289-bb78-76df37572027-kube-api-access-jlspx\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.632190 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.632162    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-tmp-dir\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.644510 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.632838    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-model-cache\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.645854 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.645830    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-home\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.646345 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.646329    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-dshm\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.646807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.646789    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35875954-d3a0-4289-bb78-76df37572027-tls-certs\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.647382 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.646240    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-home\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.647528 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.645072    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-model-cache\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.647624 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.632764    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-tmp-dir\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.658514 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.653056    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlspx\" (UniqueName: \"kubernetes.io/projected/35875954-d3a0-4289-bb78-76df37572027-kube-api-access-jlspx\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.658728 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.657896    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-dshm\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.659468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.659419    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35875954-d3a0-4289-bb78-76df37572027-tls-certs\") pod \"e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") " pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:58.795508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:58.792124    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:49:59.131478 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.131451    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"]
Nov 03 19:49:59.141928 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.141857    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:59.620439 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.620400    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"]
Nov 03 19:49:59.624204 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.624180    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.636609 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.635916    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9f46e5-openshift-default-dockercfg-4djl8\""
Nov 03 19:49:59.646504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.645525    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"]
Nov 03 19:49:59.689031 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.688992    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.689297 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.689278    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.689398 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.689384    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzdrr\" (UniqueName: \"kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-kube-api-access-wzdrr\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.689557 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.689543    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d55596f-ce16-4eab-9c20-19ea62660132-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.689675 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.689663    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.689821 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.689808    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d55596f-ce16-4eab-9c20-19ea62660132-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.689921 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.689908    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.690022 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.690007    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.690124 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.690112    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.791107 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.791060    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.791921 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.791894    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzdrr\" (UniqueName: \"kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-kube-api-access-wzdrr\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.792466 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.792440    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d55596f-ce16-4eab-9c20-19ea62660132-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.792700 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.792681    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.793229 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.793211    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d55596f-ce16-4eab-9c20-19ea62660132-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.799067 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.799044    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.799228 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.799210    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.799374 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.799355    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.803548 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.803525    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.805132 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.803951    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.808684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.794047    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d55596f-ce16-4eab-9c20-19ea62660132-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.808684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.804182    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.808684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.798991    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d55596f-ce16-4eab-9c20-19ea62660132-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.808684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.804399    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.808684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.804659    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.808684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.806739    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzdrr\" (UniqueName: \"kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-kube-api-access-wzdrr\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.808684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.808600    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.808977 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.808724    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:49:59.815302 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.815273    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb" event={"ID":"35875954-d3a0-4289-bb78-76df37572027","Type":"ContainerStarted","Data":"23022470a2279360d55d31a853593f81f8ac243bade74d33d91827110e155b99"}
Nov 03 19:49:59.815388 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.815322    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb" event={"ID":"35875954-d3a0-4289-bb78-76df37572027","Type":"ContainerStarted","Data":"d65a368f7784bee9e8932fe13e18c0e2344ba87e2ce8efa0088b856caf36411d"}
Nov 03 19:49:59.945888 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:49:59.945720    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:50:00.067078 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:00.066941    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:00.265892 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:00.265829    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb" podStartSLOduration=2.265812062 podStartE2EDuration="2.265812062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:59.839770838 +0000 UTC m=+751.994673259" watchObservedRunningTime="2026-11-03 19:50:00.265812062 +0000 UTC m=+752.420714482"
Nov 03 19:50:00.268081 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:00.268058    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"]
Nov 03 19:50:00.270912 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:50:00.270861    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d55596f_ce16_4eab_9c20_19ea62660132.slice/crio-819de2c6ce6df617730d3258ad19c1e921fc0ebebccd78933301cd91e55fac62 WatchSource:0}: Error finding container 819de2c6ce6df617730d3258ad19c1e921fc0ebebccd78933301cd91e55fac62: Status 404 returned error can't find the container with id 819de2c6ce6df617730d3258ad19c1e921fc0ebebccd78933301cd91e55fac62
Nov 03 19:50:00.273612 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:00.273579    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:00.273699 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:00.273651    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:00.273699 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:00.273690    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:00.793006 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:00.792969    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:50:00.856507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:00.845865    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg" event={"ID":"4d55596f-ce16-4eab-9c20-19ea62660132","Type":"ContainerStarted","Data":"8d61a0fce85263f9c4514723aa08f1ebbdd38137eee7b7ab08e1cb7b39348ca0"}
Nov 03 19:50:00.856507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:00.845956    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg" event={"ID":"4d55596f-ce16-4eab-9c20-19ea62660132","Type":"ContainerStarted","Data":"819de2c6ce6df617730d3258ad19c1e921fc0ebebccd78933301cd91e55fac62"}
Nov 03 19:50:00.856507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:00.852908    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:00.880580 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:00.879477    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg" podStartSLOduration=1.879455096 podStartE2EDuration="1.879455096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:00.87472168 +0000 UTC m=+753.029624095" watchObservedRunningTime="2026-11-03 19:50:00.879455096 +0000 UTC m=+753.034357516"
Nov 03 19:50:00.950131 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:00.950097    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:50:01.753510 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:01.739333    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:01.962825 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:01.962789    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:50:01.963400 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:01.963367    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg" podUID="4d55596f-ce16-4eab-9c20-19ea62660132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:50:02.463673 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:02.462142    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:02.953575 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:02.952896    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:50:02.953575 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:02.952960    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg" podUID="4d55596f-ce16-4eab-9c20-19ea62660132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:50:03.154503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:03.154151    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:03.951508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:03.947362    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:03.951508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:03.947570    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:50:03.951508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:03.947691    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg" podUID="4d55596f-ce16-4eab-9c20-19ea62660132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:50:04.677068 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:04.676983    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:04.950562 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:04.946903    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:50:04.950562 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:04.946981    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg" podUID="4d55596f-ce16-4eab-9c20-19ea62660132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:50:05.475047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:05.475007    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:05.946991 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:05.946892    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 19:50:05.947628 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:05.947590    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg" podUID="4d55596f-ce16-4eab-9c20-19ea62660132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 19:50:06.253589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:06.253552    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:07.056233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:07.056196    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:50:07.057201 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:07.057092    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:50:07.058295 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:07.058269    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 19:50:07.250035 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:07.249997    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:08.378945 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:08.378914    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:08.793150 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:08.793106    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:50:08.807301 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:08.807263    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:50:08.953243 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:08.953208    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:50:09.134866 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.134761    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:09.365180 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.360001    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"]
Nov 03 19:50:09.411444 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.411360    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.425123 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.425095    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"]
Nov 03 19:50:09.516109 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.516074    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.516293 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.516121    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.516293 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.516140    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.516293 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.516256    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.516524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.516290    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f648e3d-ef7b-4282-a072-503ecb400a96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.516524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.516398    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.516524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.516427    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.516644 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.516525    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.516644 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.516545    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khztw\" (UniqueName: \"kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-kube-api-access-khztw\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.618312 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.618261    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.618557 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.618331    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.618557 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.618357    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.618557 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.618387    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.618557 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.618433    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f648e3d-ef7b-4282-a072-503ecb400a96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.618557 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.618519    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.618854 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.618563    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.618854 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.618650    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.618854 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.618680    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khztw\" (UniqueName: \"kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-kube-api-access-khztw\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.619348 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.619309    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.619465 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.619396    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f648e3d-ef7b-4282-a072-503ecb400a96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.619623 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.619593    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.619693 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.619655    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.620038 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.620011    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.621212 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.621187    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.622247 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.622221    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.639957 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.637231    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khztw\" (UniqueName: \"kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-kube-api-access-khztw\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.639957 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.639565    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-lm9zq\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:09.727313 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:09.727272    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:10.101662 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:10.101633    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"]
Nov 03 19:50:10.103864 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:50:10.103835    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f648e3d_ef7b_4282_a072_503ecb400a96.slice/crio-f1a1cc3a85794b02f1a8018cf2eef00733594ee73e099e98811d7a6e6678b775 WatchSource:0}: Error finding container f1a1cc3a85794b02f1a8018cf2eef00733594ee73e099e98811d7a6e6678b775: Status 404 returned error can't find the container with id f1a1cc3a85794b02f1a8018cf2eef00733594ee73e099e98811d7a6e6678b775
Nov 03 19:50:10.106452 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:10.106419    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:10.106549 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:10.106501    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:10.106549 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:10.106536    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:10.302510 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:10.293956    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:10.980192 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:10.980154    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq" event={"ID":"8f648e3d-ef7b-4282-a072-503ecb400a96","Type":"ContainerStarted","Data":"f9c21bad43da7b994095ee5ebc0aa43e7b3956af6c723fffca4133335038a2fc"}
Nov 03 19:50:10.980679 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:10.980210    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq" event={"ID":"8f648e3d-ef7b-4282-a072-503ecb400a96","Type":"ContainerStarted","Data":"f1a1cc3a85794b02f1a8018cf2eef00733594ee73e099e98811d7a6e6678b775"}
Nov 03 19:50:11.008242 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:11.008191    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq" podStartSLOduration=2.008178106 podStartE2EDuration="2.008178106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:11.005898555 +0000 UTC m=+763.160801051" watchObservedRunningTime="2026-11-03 19:50:11.008178106 +0000 UTC m=+763.163080524"
Nov 03 19:50:11.090785 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:11.090748    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:11.728518 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:11.728448    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:11.730072 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:11.730044    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-lm9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:50:11.730210 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:11.730105    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq" podUID="8f648e3d-ef7b-4282-a072-503ecb400a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:50:11.990679 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:11.990581    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:12.727766 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:12.727728    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-lm9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:50:12.727959 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:12.727821    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq" podUID="8f648e3d-ef7b-4282-a072-503ecb400a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:50:12.919502 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:12.919089    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:13.727969 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:13.727932    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-lm9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:50:13.728354 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:13.727984    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq" podUID="8f648e3d-ef7b-4282-a072-503ecb400a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:50:13.988090 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:13.988003    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:14.125864 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:14.125810    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:14.728385 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:14.728350    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-lm9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:50:14.728812 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:14.728415    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq" podUID="8f648e3d-ef7b-4282-a072-503ecb400a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:50:14.808938 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:14.808902    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:15.224343 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:50:15.224280    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:50:15.728534 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:15.728471    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-lm9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:50:15.728938 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:15.728572    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq" podUID="8f648e3d-ef7b-4282-a072-503ecb400a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:50:15.944526 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:15.944466    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:16.728109 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:16.728065    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-lm9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:50:16.728322 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:16.728134    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq" podUID="8f648e3d-ef7b-4282-a072-503ecb400a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:50:16.844305 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:16.844257    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:17.545619 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:17.545580    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:17.728667 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:17.728626    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-lm9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 19:50:17.728841 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:17.728693    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq" podUID="8f648e3d-ef7b-4282-a072-503ecb400a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 19:50:18.365201 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:18.365151    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:18.983056 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:18.983020    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:18.983472 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:18.983445    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:18.984366 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:18.984340    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 19:50:19.242173 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.242091    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:19.286163 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.286118    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"]
Nov 03 19:50:19.296170 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.296142    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.301403 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.301372    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"]
Nov 03 19:50:19.369741 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.369699    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.369741 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.369737    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.370288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.369822    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.370288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.369844    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.370288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.369890    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.370288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.370010    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.370288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.370048    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8zlw\" (UniqueName: \"kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-kube-api-access-v8zlw\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.370288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.370073    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.370288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.370136    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.471370 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.471302    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.471574 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.471399    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.471574 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.471431    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8zlw\" (UniqueName: \"kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-kube-api-access-v8zlw\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.471574 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.471542    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.471755 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.471607    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.471755 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.471715    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.471755 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.471743    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.471929 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.471788    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.471929 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.471815    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.471929 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.471821    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.472831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.472136    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.472831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.472374    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.472831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.472535    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.472831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.472770    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.474427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.474397    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.474562 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.474442    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.487551 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.487527    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.487717 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.487696    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8zlw\" (UniqueName: \"kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-kube-api-access-v8zlw\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.620978 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.620885    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:19.914992 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.914967    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"]
Nov 03 19:50:19.917271 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:50:19.917228    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65ad726_8d5c_4ac2_b1f3_fe6589059ca4.slice/crio-04c7ee6bacae0dea5c3ad395bb32c3526517c5d189fd80468f21c9f45c96a4e0 WatchSource:0}: Error finding container 04c7ee6bacae0dea5c3ad395bb32c3526517c5d189fd80468f21c9f45c96a4e0: Status 404 returned error can't find the container with id 04c7ee6bacae0dea5c3ad395bb32c3526517c5d189fd80468f21c9f45c96a4e0
Nov 03 19:50:19.919811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.919776    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:19.919911 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.919854    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:19.919911 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:19.919902    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:20.051759 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:20.051705    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc" event={"ID":"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4","Type":"ContainerStarted","Data":"f5b8806ce4d79e7f1e325fcfded956cc4cae2a98d1012022f1cb9d0dc6b907fa"}
Nov 03 19:50:20.051759 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:20.051760    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc" event={"ID":"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4","Type":"ContainerStarted","Data":"04c7ee6bacae0dea5c3ad395bb32c3526517c5d189fd80468f21c9f45c96a4e0"}
Nov 03 19:50:20.075873 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:20.075811    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc" podStartSLOduration=1.075791433 podStartE2EDuration="1.075791433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:20.073371121 +0000 UTC m=+772.228273543" watchObservedRunningTime="2026-11-03 19:50:20.075791433 +0000 UTC m=+772.230693854"
Nov 03 19:50:20.234498 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:20.234449    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:20.621680 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:20.621604    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:21.059514 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:21.057784    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:21.624164 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:21.624126    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:50:21.624703 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:21.624184    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc" podUID="f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:50:21.832624 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:21.832587    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:22.077228 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:22.076788    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:22.621337 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:22.621296    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:50:22.621560 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:22.621368    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc" podUID="f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:50:23.062549 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:23.062502    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:23.621660 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:23.621625    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:50:23.621863 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:23.621699    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc" podUID="f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:50:23.766050 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:23.766010    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:24.478352 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:24.478320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:24.621300 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:24.621263    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:50:24.621474 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:24.621322    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc" podUID="f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:50:24.886722 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:24.886647    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:25.243658 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:25.241895    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:25.622216 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:25.622127    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:50:25.622216 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:25.622192    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc" podUID="f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:50:25.938603 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:25.931660    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:26.619418 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:26.619383    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:26.621478 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:26.621450    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 19:50:26.621644 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:26.621544    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc" podUID="f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 19:50:27.333876 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:27.333838    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:27.843422 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:27.843391    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:27.843837 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:27.843811    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:27.844916 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:27.844888    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 19:50:28.067185 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:28.067147    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:28.739728 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:28.739694    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:29.548259 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:29.548224    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:29.806718 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:29.806633    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:30.459927 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:30.459890    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:31.407506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:31.404068    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:32.699507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:32.697951    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:33.875858 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:33.875823    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:34.096170 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.096132    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"]
Nov 03 19:50:34.101705 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.101676    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.119207 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.119172    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"]
Nov 03 19:50:34.236198 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.236164    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.236393 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.236208    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.236393 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.236320    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.236393 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.236376    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.236585 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.236427    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.236585 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.236446    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.236585 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.236521    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793166f-3de9-4167-97d6-e3cd33c7a72f-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.236585 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.236560    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvn7s\" (UniqueName: \"kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-kube-api-access-jvn7s\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.236585 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.236579    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.337769 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.337733    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.337971 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.337807    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.337971 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.337841    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.337971 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.337868    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.337971 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.337906    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793166f-3de9-4167-97d6-e3cd33c7a72f-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.337971 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.337962    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvn7s\" (UniqueName: \"kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-kube-api-access-jvn7s\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.338230 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.337990    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.338230 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.338064    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.338230 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.338116    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.338397 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.338326    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.338567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.338545    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.339359 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.338948    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.339359 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.339127    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793166f-3de9-4167-97d6-e3cd33c7a72f-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.339536 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.339298    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.347119 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.347090    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.349588 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.349559    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"]
Nov 03 19:50:34.351931 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.351909    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.354069 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.354047    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.378253 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.378222    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"]
Nov 03 19:50:34.382005 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.381969    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvn7s\" (UniqueName: \"kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-kube-api-access-jvn7s\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.382005 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.381981    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.415812 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.415778    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:34.540417 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.540319    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dde3468-dceb-4d4e-8e71-1eba7e214145-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.540417 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.540391    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.540665 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.540547    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.540665 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.540598    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.540785 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.540709    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.540785 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.540751    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xngl8\" (UniqueName: \"kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-kube-api-access-xngl8\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.540865 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.540787    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.540908 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.540867    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.540984 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.540964    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.642360 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.642320    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.642589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.642530    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dde3468-dceb-4d4e-8e71-1eba7e214145-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.642658 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.642620    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.642728 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.642669    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.642728 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.642693    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.642728 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.642723    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.642878 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.642779    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xngl8\" (UniqueName: \"kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-kube-api-access-xngl8\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.642878 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.642818    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.642878 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.642845    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.642878 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.642862    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.643081 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.643002    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.643266 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.643239    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.643365 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.643287    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.643365 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.643312    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dde3468-dceb-4d4e-8e71-1eba7e214145-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.645111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.645088    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.645506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.645463    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.660469 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.660438    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xngl8\" (UniqueName: \"kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-kube-api-access-xngl8\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.660617 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.660438    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.707412 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.707374    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:34.716853 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.716826    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"]
Nov 03 19:50:34.721912 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:50:34.721872    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb793166f_3de9_4167_97d6_e3cd33c7a72f.slice/crio-652ce74cc3e10b06ee1889b99d4e2618f7b6cfdae752d1ed918413800dca6a4c WatchSource:0}: Error finding container 652ce74cc3e10b06ee1889b99d4e2618f7b6cfdae752d1ed918413800dca6a4c: Status 404 returned error can't find the container with id 652ce74cc3e10b06ee1889b99d4e2618f7b6cfdae752d1ed918413800dca6a4c
Nov 03 19:50:34.724782 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.724739    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:34.724870 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.724811    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:34.724870 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.724842    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:34.771555 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:34.771523    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:35.002991 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.002911    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:35.020410 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.020381    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"]
Nov 03 19:50:35.021956 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:50:35.021928    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dde3468_dceb_4d4e_8e71_1eba7e214145.slice/crio-923a036e62ba3c07dca700db556fa301cfdff1c2246540589bba5297fd68b44e WatchSource:0}: Error finding container 923a036e62ba3c07dca700db556fa301cfdff1c2246540589bba5297fd68b44e: Status 404 returned error can't find the container with id 923a036e62ba3c07dca700db556fa301cfdff1c2246540589bba5297fd68b44e
Nov 03 19:50:35.024406 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.024367    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:35.024512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.024435    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:35.024512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.024466    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:35.191506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.189213    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" event={"ID":"b793166f-3de9-4167-97d6-e3cd33c7a72f","Type":"ContainerStarted","Data":"4a9542b6b060da80a0a3933b260da69c7ddc74b5866bcb5c6d4b51a16fc70c9a"}
Nov 03 19:50:35.191506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.189317    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" event={"ID":"b793166f-3de9-4167-97d6-e3cd33c7a72f","Type":"ContainerStarted","Data":"652ce74cc3e10b06ee1889b99d4e2618f7b6cfdae752d1ed918413800dca6a4c"}
Nov 03 19:50:35.194502 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.192427    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2" event={"ID":"0dde3468-dceb-4d4e-8e71-1eba7e214145","Type":"ContainerStarted","Data":"ee578c20e6a5cfbaebcede3bd07d17398fb12d4d1999375500d3b9ef87e29c1e"}
Nov 03 19:50:35.194502 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.192532    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2" event={"ID":"0dde3468-dceb-4d4e-8e71-1eba7e214145","Type":"ContainerStarted","Data":"923a036e62ba3c07dca700db556fa301cfdff1c2246540589bba5297fd68b44e"}
Nov 03 19:50:35.213120 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.213058    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" podStartSLOduration=1.213041191 podStartE2EDuration="1.213041191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:35.212367765 +0000 UTC m=+787.367270186" watchObservedRunningTime="2026-11-03 19:50:35.213041191 +0000 UTC m=+787.367943615"
Nov 03 19:50:35.242105 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.242040    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2" podStartSLOduration=1.242019429 podStartE2EDuration="1.242019429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:35.237461764 +0000 UTC m=+787.392364183" watchObservedRunningTime="2026-11-03 19:50:35.242019429 +0000 UTC m=+787.396921859"
Nov 03 19:50:35.416384 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.416302    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:35.417863 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.417839    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 19:50:35.417984 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.417883    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" podUID="b793166f-3de9-4167-97d6-e3cd33c7a72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 19:50:35.640266 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.640071    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:35.708111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:35.708078    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:36.417739 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:36.417707    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 19:50:36.418276 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:36.417760    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" podUID="b793166f-3de9-4167-97d6-e3cd33c7a72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 19:50:36.672552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:36.672384    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:36.711159 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:36.711125    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:50:36.711343 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:36.711180    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2" podUID="0dde3468-dceb-4d4e-8e71-1eba7e214145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 19:50:37.416502 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:37.416359    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 19:50:37.416502 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:37.416436    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" podUID="b793166f-3de9-4167-97d6-e3cd33c7a72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 19:50:37.496579 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:37.496468    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:37.708708 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:37.708673    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:50:37.708868 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:37.708734    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2" podUID="0dde3468-dceb-4d4e-8e71-1eba7e214145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 19:50:38.416076 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:38.416041    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 19:50:38.416284 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:38.416126    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" podUID="b793166f-3de9-4167-97d6-e3cd33c7a72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 19:50:38.564137 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:38.564091    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:38.708972 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:38.708936    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:50:38.709255 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:38.709224    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2" podUID="0dde3468-dceb-4d4e-8e71-1eba7e214145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 19:50:39.416502 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:39.416456    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 19:50:39.416702 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:39.416533    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" podUID="b793166f-3de9-4167-97d6-e3cd33c7a72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 19:50:39.568506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:39.568451    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:39.707846 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:39.707809    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:50:39.708049 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:39.707888    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2" podUID="0dde3468-dceb-4d4e-8e71-1eba7e214145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 19:50:40.205305 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:40.205274    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:40.420595 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:40.420028    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 19:50:40.420595 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:40.420095    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" podUID="b793166f-3de9-4167-97d6-e3cd33c7a72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 19:50:40.707906 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:40.707874    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:50:40.708370 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:40.707941    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2" podUID="0dde3468-dceb-4d4e-8e71-1eba7e214145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 19:50:40.771505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:40.767846    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:41.013021 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:41.012931    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"]
Nov 03 19:50:41.094151 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:41.093982    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:41.417191 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:41.417093    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 19:50:41.417191 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:41.417179    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" podUID="b793166f-3de9-4167-97d6-e3cd33c7a72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 19:50:41.746377 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:41.746332    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:42.092014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:42.091903    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:42.092939 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:42.092910    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:42.094885 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:42.094865    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:50:42.448842 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:42.448713    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:43.221737 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:43.221706    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:43.459612 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:43.459574    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:50:43.459912 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:43.459883    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" podUID="b793166f-3de9-4167-97d6-e3cd33c7a72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:43.564633 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:43.564553    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:43.566221 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:43.566197    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:43.569469 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:43.569442    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 19:50:44.034341 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:44.034297    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:44.754148 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:44.754103    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:45.218775 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:50:45.218734    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:50:45.652766 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:45.652686    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:46.578059 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:46.578024    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:47.352935 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:47.352899    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:48.204349 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:48.204314    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:48.937034 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:48.936995    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:49.102525 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.102460    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"]
Nov 03 19:50:49.109846 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.109820    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.131764 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.131734    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"]
Nov 03 19:50:49.174349 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.174314    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:49.191402 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.191308    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.191402 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.191354    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.191402 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.191379    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.191631 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.191551    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzvr\" (UniqueName: \"kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-kube-api-access-smzvr\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.191692 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.191665    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.191741 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.191726    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.191828 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.191811    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.191887 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.191849    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.191982 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.191969    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.293201 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.293159    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.293765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.293295    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.293765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.293326    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.293765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.293349    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.293765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.293394    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzvr\" (UniqueName: \"kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-kube-api-access-smzvr\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.293765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.293441    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.293765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.293496    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.293765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.293540    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.293765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.293571    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.294221 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.293806    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.294221 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.293876    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.294221 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.294159    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.294221 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.294172    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.294907 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.294885    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.296695 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.296665    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.296793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.296710    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.307275 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.307170    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.307275 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.307231    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzvr\" (UniqueName: \"kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-kube-api-access-smzvr\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.309054 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.309030    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:49.445554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.434540    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:49.818622 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.818591    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"]
Nov 03 19:50:49.819568 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:50:49.819532    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca61f7f_bcc7_4b50_83fc_3b38456848d9.slice/crio-9f10395afeb3978ef8275de8f1c452523c4a5c179e3e6950d796acb9721206ba WatchSource:0}: Error finding container 9f10395afeb3978ef8275de8f1c452523c4a5c179e3e6950d796acb9721206ba: Status 404 returned error can't find the container with id 9f10395afeb3978ef8275de8f1c452523c4a5c179e3e6950d796acb9721206ba
Nov 03 19:50:49.821973 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.821934    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:49.822053 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.821999    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:49.822053 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:49.822029    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:50:50.166917 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:50.166814    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:50.359505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:50.351613    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" event={"ID":"7ca61f7f-bcc7-4b50-83fc-3b38456848d9","Type":"ContainerStarted","Data":"47211acbfed1630cbff05bbbc8c644eff51e1827e3a9bbb58a8d5c52500cb8b6"}
Nov 03 19:50:50.359505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:50.351689    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" event={"ID":"7ca61f7f-bcc7-4b50-83fc-3b38456848d9","Type":"ContainerStarted","Data":"9f10395afeb3978ef8275de8f1c452523c4a5c179e3e6950d796acb9721206ba"}
Nov 03 19:50:50.377880 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:50.377830    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" podStartSLOduration=1.377815446 podStartE2EDuration="1.377815446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:50.376051611 +0000 UTC m=+802.530954035" watchObservedRunningTime="2026-11-03 19:50:50.377815446 +0000 UTC m=+802.532717865"
Nov 03 19:50:50.453861 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:50.445845    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:50.459507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:50.455606    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:50:50.459507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:50.455683    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" podUID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:50:50.993268 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:50.993169    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:51.444991 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:51.444900    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:50:51.445592 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:51.445557    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" podUID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:50:51.774102 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:51.774063    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:52.445800 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:52.445760    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:50:52.446411 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:52.446375    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" podUID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:50:52.561067 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:52.561028    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:53.255150 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:53.255109    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:53.437987 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:53.437948    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:50:53.439268 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:53.439230    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" podUID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:50:54.160674 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.160627    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:54.263570 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.262849    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"]
Nov 03 19:50:54.275359 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.275324    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.284382 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.280153    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"distinct-w1-32e482-kserve-self-signed-certs\""
Nov 03 19:50:54.284382 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.281210    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"]
Nov 03 19:50:54.347514 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.347380    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:54.382960 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.382897    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-model-cache\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.383168 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.382969    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-home\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.383168 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.382997    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-dshm\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.383168 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.383065    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-tmp-dir\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.383168 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.383124    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcldh\" (UniqueName: \"kubernetes.io/projected/92ffe4fd-3411-41b3-898f-44da3191ee57-kube-api-access-rcldh\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.383403 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.383212    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ffe4fd-3411-41b3-898f-44da3191ee57-tls-certs\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.438583 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.437940    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:50:54.438583 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.438018    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" podUID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:50:54.491523 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.489673    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-tmp-dir\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.491523 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.489761    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcldh\" (UniqueName: \"kubernetes.io/projected/92ffe4fd-3411-41b3-898f-44da3191ee57-kube-api-access-rcldh\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.491523 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.489843    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ffe4fd-3411-41b3-898f-44da3191ee57-tls-certs\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.491523 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.489967    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-model-cache\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.491523 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.490019    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-home\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.491523 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.490053    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-dshm\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.491523 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.491366    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-model-cache\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.492159 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.491721    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-tmp-dir\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.503560 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.492289    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-home\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.514503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.513119    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ffe4fd-3411-41b3-898f-44da3191ee57-tls-certs\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.517649 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.517620    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcldh\" (UniqueName: \"kubernetes.io/projected/92ffe4fd-3411-41b3-898f-44da3191ee57-kube-api-access-rcldh\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.528813 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.526590    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-dshm\") pod \"distinct-w1-32e482-kserve-5b7d5b55db-6m4gf\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.602042 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.601579    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:54.920719 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:54.920690    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"]
Nov 03 19:50:55.063424 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:55.063382    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:55.397611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:55.397521    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" event={"ID":"92ffe4fd-3411-41b3-898f-44da3191ee57","Type":"ContainerStarted","Data":"9c5473d2d956c8eac1e6ed21263c2deaab3911e14804a6104c210aa0938f23c6"}
Nov 03 19:50:55.397611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:55.397566    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" event={"ID":"92ffe4fd-3411-41b3-898f-44da3191ee57","Type":"ContainerStarted","Data":"e6874fc5cd449a9c2675f5ad61524749f7178b297f3e2568da7b2266d8baf436"}
Nov 03 19:50:55.417056 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:55.417000    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podStartSLOduration=1.416986951 podStartE2EDuration="1.416986951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:55.416435376 +0000 UTC m=+807.571337808" watchObservedRunningTime="2026-11-03 19:50:55.416986951 +0000 UTC m=+807.571889410"
Nov 03 19:50:55.454507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:55.454219    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:50:55.454507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:55.454283    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" podUID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:50:55.602148 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:55.602116    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:50:55.968544 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:55.968506    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:56.086784 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.086735    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb" podUID="35875954-d3a0-4289-bb78-76df37572027" containerName="main" containerID="cri-o://23022470a2279360d55d31a853593f81f8ac243bade74d33d91827110e155b99" gracePeriod=45
Nov 03 19:50:56.357060 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.357029    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:50:56.406955 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.406918    5659 generic.go:356] "Generic (PLEG): container finished" podID="35875954-d3a0-4289-bb78-76df37572027" containerID="23022470a2279360d55d31a853593f81f8ac243bade74d33d91827110e155b99" exitCode=0
Nov 03 19:50:56.407664 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.407645    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"
Nov 03 19:50:56.407892 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.407872    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb" event={"ID":"35875954-d3a0-4289-bb78-76df37572027","Type":"ContainerDied","Data":"23022470a2279360d55d31a853593f81f8ac243bade74d33d91827110e155b99"}
Nov 03 19:50:56.407958 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.407910    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb" event={"ID":"35875954-d3a0-4289-bb78-76df37572027","Type":"ContainerDied","Data":"d65a368f7784bee9e8932fe13e18c0e2344ba87e2ce8efa0088b856caf36411d"}
Nov 03 19:50:56.407958 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.407938    5659 scope.go:122] "RemoveContainer" containerID="23022470a2279360d55d31a853593f81f8ac243bade74d33d91827110e155b99"
Nov 03 19:50:56.420475 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.420448    5659 scope.go:122] "RemoveContainer" containerID="23022470a2279360d55d31a853593f81f8ac243bade74d33d91827110e155b99"
Nov 03 19:50:56.420858 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:50:56.420820    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23022470a2279360d55d31a853593f81f8ac243bade74d33d91827110e155b99\": container with ID starting with 23022470a2279360d55d31a853593f81f8ac243bade74d33d91827110e155b99 not found: ID does not exist" containerID="23022470a2279360d55d31a853593f81f8ac243bade74d33d91827110e155b99"
Nov 03 19:50:56.420951 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.420868    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23022470a2279360d55d31a853593f81f8ac243bade74d33d91827110e155b99"} err="failed to get container status \"23022470a2279360d55d31a853593f81f8ac243bade74d33d91827110e155b99\": rpc error: code = NotFound desc = could not find container \"23022470a2279360d55d31a853593f81f8ac243bade74d33d91827110e155b99\": container with ID starting with 23022470a2279360d55d31a853593f81f8ac243bade74d33d91827110e155b99 not found: ID does not exist"
Nov 03 19:50:56.436261 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.435805    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:50:56.436261 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.435861    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" podUID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 19:50:56.514948 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.514912    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35875954-d3a0-4289-bb78-76df37572027-tls-certs\" (UniqueName: \"kubernetes.io/secret/35875954-d3a0-4289-bb78-76df37572027-tls-certs\") pod \"35875954-d3a0-4289-bb78-76df37572027\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") "
Nov 03 19:50:56.515146 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.514971    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35875954-d3a0-4289-bb78-76df37572027-kube-api-access-jlspx\" (UniqueName: \"kubernetes.io/projected/35875954-d3a0-4289-bb78-76df37572027-kube-api-access-jlspx\") pod \"35875954-d3a0-4289-bb78-76df37572027\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") "
Nov 03 19:50:56.515146 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.515057    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-dshm\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-dshm\") pod \"35875954-d3a0-4289-bb78-76df37572027\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") "
Nov 03 19:50:56.515146 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.515115    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-tmp-dir\") pod \"35875954-d3a0-4289-bb78-76df37572027\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") "
Nov 03 19:50:56.515297 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.515185    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-home\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-home\") pod \"35875954-d3a0-4289-bb78-76df37572027\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") "
Nov 03 19:50:56.515297 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.515261    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-model-cache\") pod \"35875954-d3a0-4289-bb78-76df37572027\" (UID: \"35875954-d3a0-4289-bb78-76df37572027\") "
Nov 03 19:50:56.515419 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.515361    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-tmp-dir" pod "35875954-d3a0-4289-bb78-76df37572027" (UID: "35875954-d3a0-4289-bb78-76df37572027"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:56.515584 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.515536    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-model-cache" pod "35875954-d3a0-4289-bb78-76df37572027" (UID: "35875954-d3a0-4289-bb78-76df37572027"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:56.515942 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.515921    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-home" pod "35875954-d3a0-4289-bb78-76df37572027" (UID: "35875954-d3a0-4289-bb78-76df37572027"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:56.516104 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.516082    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:56.516216 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.516107    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:56.518547 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.517839    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35875954-d3a0-4289-bb78-76df37572027-kube-api-access-jlspx" pod "35875954-d3a0-4289-bb78-76df37572027" (UID: "35875954-d3a0-4289-bb78-76df37572027"). InnerVolumeSpecName "kube-api-access-jlspx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:50:56.518547 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.517945    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35875954-d3a0-4289-bb78-76df37572027-tls-certs" pod "35875954-d3a0-4289-bb78-76df37572027" (UID: "35875954-d3a0-4289-bb78-76df37572027"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:50:56.518547 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.518021    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-dshm" pod "35875954-d3a0-4289-bb78-76df37572027" (UID: "35875954-d3a0-4289-bb78-76df37572027"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:56.617792 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.617756    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:56.617792 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.617789    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35875954-d3a0-4289-bb78-76df37572027-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:56.617792 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.617801    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35875954-d3a0-4289-bb78-76df37572027-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:56.618054 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.617815    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlspx\" (UniqueName: \"kubernetes.io/projected/35875954-d3a0-4289-bb78-76df37572027-kube-api-access-jlspx\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:56.732674 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.732640    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"]
Nov 03 19:50:56.738848 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.738818    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/e2e-conflict-a-a0f38071-kserve-5b549fcdf7-kgcpb"]
Nov 03 19:50:56.757164 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:56.757135    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:57.440086 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:57.440053    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:57.440554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:57.440538    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:57.441371 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:57.441346    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 19:50:57.696340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:57.696260    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:57.979516 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:57.979462    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35875954-d3a0-4289-bb78-76df37572027" path="/var/lib/kubelet/pods/35875954-d3a0-4289-bb78-76df37572027/volumes"
Nov 03 19:50:58.930232 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:58.930043    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:59.246320 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.246277    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"]
Nov 03 19:50:59.247084 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.247059    5659 cpu_manager.go:414] "containerMap: removing container" podUID="35875954-d3a0-4289-bb78-76df37572027" containerName="main"
Nov 03 19:50:59.247211 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.247086    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35875954-d3a0-4289-bb78-76df37572027" containerName="main"
Nov 03 19:50:59.247280 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.247253    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="35875954-d3a0-4289-bb78-76df37572027" containerName="main"
Nov 03 19:50:59.252236 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.252210    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.256242 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.256218    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2b5703-openshift-default-dockercfg-jmjbm\""
Nov 03 19:50:59.268197 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.265934    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"]
Nov 03 19:50:59.298504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.280399    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.298504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.280525    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.298504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.280578    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.298504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.280646    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.298504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.280701    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.298504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.281392    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.298504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.281549    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.298504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.281685    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbbn\" (UniqueName: \"kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-kube-api-access-xlbbn\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.298504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.281760    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.383131    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbbn\" (UniqueName: \"kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-kube-api-access-xlbbn\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.383217    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.383847    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.383894    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.383922    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.383971    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.383992    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.384020    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.384079    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.384700    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.384959    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.385188    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.385361    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.386567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.385905    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.388210 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.388186    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.391430 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.391407    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.396277 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.396255    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbbn\" (UniqueName: \"kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-kube-api-access-xlbbn\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.399275 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.399250    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:50:59.610456 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:50:59.610301    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:51:00.130784 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:00.130754    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"]
Nov 03 19:51:00.172528 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:00.146333    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:00.172528 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:00.171292    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:00.172528 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:00.171363    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:00.218921 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:00.218890    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:00.595516 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:00.594633    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" event={"ID":"38cd3425-f497-4177-a1ae-3e8dd72a8c16","Type":"ContainerStarted","Data":"cc5de8de49a6da238ce48e3bf5f8c610544109cd7bd2d5ce6f58c4646b18dfa5"}
Nov 03 19:51:00.595516 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:00.594703    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" event={"ID":"38cd3425-f497-4177-a1ae-3e8dd72a8c16","Type":"ContainerStarted","Data":"69988785659dd1f6bde174e1748cd1511fda6693cff00720bdf87bc68f61ca60"}
Nov 03 19:51:00.612531 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:00.610509    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:51:00.979857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:00.979516    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnfsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:51:00.979857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:00.979583    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" podUID="9fa9e0c2-1bfa-4b41-b033-8d286b4c63be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:01.550444 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:01.550316    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:01.612130 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:01.612094    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:51:01.612327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:01.612156    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 19:51:02.372504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:02.372277    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:02.611301 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:02.611265    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:51:02.611826 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:02.611338    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 19:51:03.050025 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:03.049943    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:03.611733 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:03.611700    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:51:03.612120 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:03.611769    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 19:51:03.750370 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:03.750338    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:04.370503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.369440    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" podStartSLOduration=5.369418156 podStartE2EDuration="5.369418156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:00.640795446 +0000 UTC m=+812.795697866" watchObservedRunningTime="2026-11-03 19:51:04.369418156 +0000 UTC m=+816.524320576"
Nov 03 19:51:04.370503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.369819    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"]
Nov 03 19:51:04.393530 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.388971    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.395548 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.395051    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"]
Nov 03 19:51:04.449767 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.449730    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:04.496058 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.496024    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.496355 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.496337    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.496457 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.496448    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d34e5e5-88b0-4040-9353-840bd249d790-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.496580 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.496560    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.496758 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.496734    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d34e5e5-88b0-4040-9353-840bd249d790-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.496875 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.496857    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.496975 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.496943    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bddwl\" (UniqueName: \"kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-kube-api-access-bddwl\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.497046 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.497009    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.497131 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.497098    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.598588 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.598552    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.598780 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.598617    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.598780 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.598661    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.598780 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.598696    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.598780 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.598714    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d34e5e5-88b0-4040-9353-840bd249d790-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.598780 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.598734    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.598780 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.598753    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d34e5e5-88b0-4040-9353-840bd249d790-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.599093 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.598808    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.599093 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.598837    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddwl\" (UniqueName: \"kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-kube-api-access-bddwl\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.599190 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.599100    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.599300 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.599266    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.599461 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.599393    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.599581 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.599527    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.599726 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.599705    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d34e5e5-88b0-4040-9353-840bd249d790-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.601310 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.601293    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.601703 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.601682    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d34e5e5-88b0-4040-9353-840bd249d790-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.601783 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.601770    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:51:04.612847 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.612804    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:51:04.613382 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.612855    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 19:51:04.618713 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.617328    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:51:04.619388 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.619297    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.622016 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.621827    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddwl\" (UniqueName: \"kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-kube-api-access-bddwl\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.695992 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.691972    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"]
Nov 03 19:51:04.698739 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.697584    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"]
Nov 03 19:51:04.698739 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.697660    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:51:04.698739 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.697750    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.707075 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.705984    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"distinct-2-w2-2b5703-kserve-self-signed-certs\""
Nov 03 19:51:04.717840 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.717812    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:04.744509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.741580    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:04.810362 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.802415    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-dshm\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.811747 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.811718    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-home\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.812187 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.811998    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tls-certs\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.812470 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.812446    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-model-cache\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.812814 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.812792    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkvqw\" (UniqueName: \"kubernetes.io/projected/f2d9f7b7-51af-4be2-b959-3143ebc16f58-kube-api-access-nkvqw\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.813413 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.813393    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tmp-dir\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.925575 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.920176    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-model-cache\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.925575 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.920333    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkvqw\" (UniqueName: \"kubernetes.io/projected/f2d9f7b7-51af-4be2-b959-3143ebc16f58-kube-api-access-nkvqw\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.925575 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.920403    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tmp-dir\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.925575 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.920445    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-dshm\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.925575 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.920496    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-home\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.925575 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.920569    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tls-certs\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.925575 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.921720    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tmp-dir\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.925575 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.922006    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-model-cache\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.933507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.930862    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-home\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.969692 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.969620    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkvqw\" (UniqueName: \"kubernetes.io/projected/f2d9f7b7-51af-4be2-b959-3143ebc16f58-kube-api-access-nkvqw\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.979016 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.969976    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tls-certs\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:04.979016 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:04.970215    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-dshm\") pod \"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:05.067511 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.066804    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:05.277927 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.276458    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"]
Nov 03 19:51:05.279420 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.278939    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:05.279420 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.279033    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:05.279420 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.279062    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:05.353002 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.352917    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:05.441563 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.439295    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"]
Nov 03 19:51:05.611525 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.611010    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:51:05.611525 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.611057    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 19:51:05.639627 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.639543    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" event={"ID":"f2d9f7b7-51af-4be2-b959-3143ebc16f58","Type":"ContainerStarted","Data":"79a6a31d347d053fdcc2425aa4f98e70f5def05756a62dfd189a6bcbb02a1103"}
Nov 03 19:51:05.639627 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.639601    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" event={"ID":"f2d9f7b7-51af-4be2-b959-3143ebc16f58","Type":"ContainerStarted","Data":"edc4473cced726887611d9752390f3308e25b56f029c0373b9cecbfa171e1042"}
Nov 03 19:51:05.651510 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.644176    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" event={"ID":"1d34e5e5-88b0-4040-9353-840bd249d790","Type":"ContainerStarted","Data":"21abac9cea58740d3e74430a303aae277c5870c67bf42a66de51adc0dfb8ee2a"}
Nov 03 19:51:05.651510 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.644261    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" event={"ID":"1d34e5e5-88b0-4040-9353-840bd249d790","Type":"ContainerStarted","Data":"20ad5b2695ed1a6358e5f8612160076c693e5654653c1877b35a93f137821bae"}
Nov 03 19:51:05.691532 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.691130    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" podStartSLOduration=1.6911101 podStartE2EDuration="1.6911101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:05.689957425 +0000 UTC m=+817.844859852" watchObservedRunningTime="2026-11-03 19:51:05.6911101 +0000 UTC m=+817.846012520"
Nov 03 19:51:05.700556 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.694951    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podStartSLOduration=1.6949322759999998 podStartE2EDuration="1.694932276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:05.664269386 +0000 UTC m=+817.819171806" watchObservedRunningTime="2026-11-03 19:51:05.694932276 +0000 UTC m=+817.849834696"
Nov 03 19:51:05.723503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:05.718254    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:06.089618 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:06.084583    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:06.555376 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:06.554328    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:06.614964 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:06.614923    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:51:06.615241 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:06.615211    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 19:51:06.753067 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:06.753030    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:51:06.764772 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:06.764734    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:51:07.251553 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:07.250705    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:51:07.328506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:07.327669    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:07.429679 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:07.428576    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:07.533943 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:07.533582    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:07.590924 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:07.590606    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:07.618508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:07.610957    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:51:07.618508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:07.611019    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 19:51:07.631515 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:07.630113    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:07.733036 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:07.732586    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:51:07.733036 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:07.732645    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:51:08.073386 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:08.028776    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:08.271503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:08.251440    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:08.271503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:08.252013    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:51:08.273617 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:08.273187    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:08.360497 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:08.342416    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:08.360497 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:08.349008    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:51:08.619615 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:08.619525    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:51:08.620166 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:08.620136    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 19:51:08.690407 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:08.690370    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:08.754503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:08.747235    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:51:08.756263 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:08.756220    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:51:08.852650 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:08.852572    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:51:09.187297 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:09.187264    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:51:09.589539 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:09.589341    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:09.618714 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:09.618679    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:51:09.618999 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:09.618968    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Nov 03 19:51:09.726421 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:09.725929    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:51:09.726421 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:09.725997    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:51:09.882271 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:09.882180    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:10.675507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:10.671008    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:10.720970 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:10.720923    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:51:10.721276 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:10.721228    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:51:11.113835 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:11.113793    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:51:11.139914 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:11.116364    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:51:11.164102 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:11.142536    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 19:51:11.358504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:11.357368    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:11.476525 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:11.471584    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:11.674649 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:11.672595    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:51:11.674649 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:11.673208    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:51:11.799226 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:11.797817    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:51:11.799226 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:11.797875    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:51:11.857869 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:11.857620    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:12.016629 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:12.016598    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:51:12.268061 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:12.268028    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:12.720131 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:12.719696    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:51:12.720131 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:12.719771    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:51:13.128344 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:13.128244    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:13.718765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:13.718727    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:51:13.718943 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:13.718795    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 19:51:13.826682 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:13.826646    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:14.542543 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:14.542502    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:14.981732 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:14.981698    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:14.982174 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:14.982148    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:14.983167 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:14.983145    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:51:15.083622 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:15.083587    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:15.097076 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:15.097050    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:15.203273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:15.203242    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:51:15.218436 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:51:15.218390    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:51:15.428231 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:15.428143    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:16.152206 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:16.152166    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:17.326402 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:17.326366    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:18.440349 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:18.440311    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:18.625538 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:18.625440    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:19.576011 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:19.575972    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:20.358622 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:20.358581    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:22.586436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:22.586374    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:22.716705 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:22.716676    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:23.338619 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:23.338586    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:24.036696 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:24.036656    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:24.740461 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:24.740418    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:25.525950 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:25.525922    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:26.531800 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:26.531767    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:26.676272 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:26.676231    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:26.815869 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:26.814324    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:27.661832 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:27.661793    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:28.461675 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:28.461633    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:29.151988 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:29.151954    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:29.948075 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:29.948034    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:30.068942 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:30.068907    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:30.748649 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:30.748596    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:30.953945 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:30.953896    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:31.447376 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:31.447346    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:32.149674 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:32.149637    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:33.036155 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:33.036119    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:34.044053 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:34.044017    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:35.064508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:35.064087    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:35.196064 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:35.196027    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:35.322969 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:35.322857    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:35.974298 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:35.974266    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:36.636860 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:36.636824    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:37.338123 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:37.338092    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:38.047747 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:38.047710    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:38.613529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:38.613498    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:38.764409 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:38.764376    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:39.017845 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:39.017815    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:39.762950 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:39.762915    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:40.744682 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:40.744651    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:41.438879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:41.438848    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:42.238280 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.238234    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:42.482024 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.481984    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"]
Nov 03 19:51:42.488196 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.488166    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.493172 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.493110    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"distinct-w3-9f46e5-kserve-self-signed-certs\""
Nov 03 19:51:42.493540 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.493347    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"kube-root-ca.crt\""
Nov 03 19:51:42.493763 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.493412    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"openshift-service-ca.crt\""
Nov 03 19:51:42.494589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.493472    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"default-dockercfg-zrxnc\""
Nov 03 19:51:42.497156 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.497131    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"]
Nov 03 19:51:42.560951 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.560920    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:42.643813 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.643766    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgmw\" (UniqueName: \"kubernetes.io/projected/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-kube-api-access-sxgmw\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.646028 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.643834    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-model-cache\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.650851 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.650821    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tmp-dir\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.651515 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.651471    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tls-certs\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.651736 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.651712    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-home\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.651864 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.651801    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-dshm\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.752800 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.752674    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-home\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.752975 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.752803    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-dshm\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.752975 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.752937    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgmw\" (UniqueName: \"kubernetes.io/projected/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-kube-api-access-sxgmw\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.753094 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.752973    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-model-cache\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.753094 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.753034    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tmp-dir\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.753208 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.753093    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tls-certs\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.753208 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.753149    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-home\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.753434 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.753410    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-model-cache\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.753624 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.753543    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tmp-dir\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.755823 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.755797    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-dshm\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.756114 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.756093    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tls-certs\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.768113 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.768085    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgmw\" (UniqueName: \"kubernetes.io/projected/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-kube-api-access-sxgmw\") pod \"distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:42.805172 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:42.804950    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:43.102718 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:43.102690    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"]
Nov 03 19:51:43.105610 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:51:43.105586    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11abc9df_4f9d_4e6b_b1cf_5f2e64db0b4a.slice/crio-eee72930bdba7d583eb95c29121c4c39261536fabecba7d1871319f9f910a2dc WatchSource:0}: Error finding container eee72930bdba7d583eb95c29121c4c39261536fabecba7d1871319f9f910a2dc: Status 404 returned error can't find the container with id eee72930bdba7d583eb95c29121c4c39261536fabecba7d1871319f9f910a2dc
Nov 03 19:51:43.136402 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:43.136379    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:43.676292 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:43.676258    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" event={"ID":"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a","Type":"ContainerStarted","Data":"90cf9d1e6a36dacb86298d9be5e5cf8cfcb1f4c6cf602fa95ff7bb47cb5de4d3"}
Nov 03 19:51:43.676782 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:43.676300    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" event={"ID":"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a","Type":"ContainerStarted","Data":"eee72930bdba7d583eb95c29121c4c39261536fabecba7d1871319f9f910a2dc"}
Nov 03 19:51:43.696652 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:43.696598    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podStartSLOduration=1.696582729 podStartE2EDuration="1.696582729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:43.694472366 +0000 UTC m=+855.849374786" watchObservedRunningTime="2026-11-03 19:51:43.696582729 +0000 UTC m=+855.851485147"
Nov 03 19:51:43.805103 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:43.805073    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:43.849929 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:43.849880    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:44.764040 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:44.764003    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:45.223634 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:51:45.223592    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:51:45.552455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:45.552378    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:46.273803 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:46.273772    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:47.062741 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:47.062699    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:47.867197 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:47.867164    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:50.017413 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:50.017309    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:50.983507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:50.982577    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:52.523426 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:52.523372    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:52.810519 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:52.809552    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:52.829595 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:52.829560    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:53.003506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:53.003218    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:51:54.105128 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.105096    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:54.395168 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.395091    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"]
Nov 03 19:51:54.399475 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.399449    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.405109 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.405084    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"]
Nov 03 19:51:54.405693 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.405671    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3d0df\"/\"distinct-w0-d3d0df-kserve-self-signed-certs\""
Nov 03 19:51:54.441585 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.441552    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-home\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.441791 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.441617    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-model-cache\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.441791 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.441742    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66h52\" (UniqueName: \"kubernetes.io/projected/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-kube-api-access-66h52\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.441897 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.441819    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tls-certs\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.441982 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.441963    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tmp-dir\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.442066 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.442003    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-dshm\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.467606 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.467567    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:54.544543 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.544498    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66h52\" (UniqueName: \"kubernetes.io/projected/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-kube-api-access-66h52\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.544909 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.544884    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tls-certs\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.545154 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.545135    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tmp-dir\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.545293 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.545277    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-dshm\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.545401 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.545385    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-home\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.545533 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.545516    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-model-cache\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.546311 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.546113    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-model-cache\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.546838 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.546814    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tmp-dir\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.547638 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.547575    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-home\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.565002 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.564969    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tls-certs\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.565474 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.565449    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-dshm\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.571970 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.571252    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66h52\" (UniqueName: \"kubernetes.io/projected/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-kube-api-access-66h52\") pod \"distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.715807 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.715770    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:54.996958 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:54.996926    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"]
Nov 03 19:51:54.997076 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:51:54.996926    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e5e3878_4eae_4f9e_8ae4_9483719cfbda.slice/crio-930bfd25721f6cf7446e4120d5d55fe0a362cf8eef45fe389e27778d9a979b03 WatchSource:0}: Error finding container 930bfd25721f6cf7446e4120d5d55fe0a362cf8eef45fe389e27778d9a979b03: Status 404 returned error can't find the container with id 930bfd25721f6cf7446e4120d5d55fe0a362cf8eef45fe389e27778d9a979b03
Nov 03 19:51:55.263292 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:55.263193    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:55.948098 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:55.948062    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" event={"ID":"6e5e3878-4eae-4f9e-8ae4-9483719cfbda","Type":"ContainerStarted","Data":"a3607cb980e7c9ebf6ba92135b100a6fcc54ea0c0cc7b734dc9c22e758b71af5"}
Nov 03 19:51:55.948098 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:55.948105    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" event={"ID":"6e5e3878-4eae-4f9e-8ae4-9483719cfbda","Type":"ContainerStarted","Data":"930bfd25721f6cf7446e4120d5d55fe0a362cf8eef45fe389e27778d9a979b03"}
Nov 03 19:51:55.970457 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:55.970401    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podStartSLOduration=1.9703803199999999 podStartE2EDuration="1.97038032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:55.9673094 +0000 UTC m=+868.122211821" watchObservedRunningTime="2026-11-03 19:51:55.97038032 +0000 UTC m=+868.125282740"
Nov 03 19:51:56.049840 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:56.049799    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:56.716475 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:56.716431    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:51:56.923604 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:56.923572    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:57.650644 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:57.650606    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:58.462148 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:58.462119    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:59.159731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.159699    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:59.354056 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.354022    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"]
Nov 03 19:51:59.357850 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.357827    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.374396 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.374366    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"]
Nov 03 19:51:59.388361 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.387744    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"]
Nov 03 19:51:59.392294 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.392265    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.408681 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.408357    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"]
Nov 03 19:51:59.414576 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.414476    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.414576 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.414562    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c95cd0-cf66-4a36-9c3e-f917ee445790-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.414773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.414592    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.414773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.414691    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.414773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.414709    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8pgr\" (UniqueName: \"kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-kube-api-access-k8pgr\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.414773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.414736    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.414773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.414766    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.415054 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.414784    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.415054 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.414807    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.516369 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.516336    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.516793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.516386    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.516793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.516461    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.516793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.516560    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.516793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.516613    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.516793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.516647    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.516793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.516684    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.516793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.516724    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b7fd36-49dc-44d2-8b0b-284713245655-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.517205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.516807    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.517205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.516823    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.517205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.516831    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.517205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.516965    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.517205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.516996    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c95cd0-cf66-4a36-9c3e-f917ee445790-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.517205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.517020    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.517205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.517063    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.517205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.517114    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.517205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.517155    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knsfv\" (UniqueName: \"kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-kube-api-access-knsfv\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.517724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.517229    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.517724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.517256    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8pgr\" (UniqueName: \"kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-kube-api-access-k8pgr\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.517724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.517274    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.517724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.517280    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.517724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.517393    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.517724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.517695    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c95cd0-cf66-4a36-9c3e-f917ee445790-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.518951 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.518929    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.519255 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.519235    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.529524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.529504    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.529981 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.529961    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8pgr\" (UniqueName: \"kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-kube-api-access-k8pgr\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.618664 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.618627    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.618884 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.618686    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.618884 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.618722    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.618884 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.618755    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b7fd36-49dc-44d2-8b0b-284713245655-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.618884 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.618829    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.619165 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.618927    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.619165 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.618978    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.619165 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.619014    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsfv\" (UniqueName: \"kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-kube-api-access-knsfv\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.619165 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.619070    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.619165 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.619082    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.619165 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.619104    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.619477 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.619457    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.619554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.619473    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.619554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.619536    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b7fd36-49dc-44d2-8b0b-284713245655-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.621431 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.621408    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.621743 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.621725    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.634540 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.634506    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.634669 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.634631    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knsfv\" (UniqueName: \"kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-kube-api-access-knsfv\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.679534 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.679420    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:51:59.708173 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.708142    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:51:59.943734 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.943625    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:59.980900 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.980099    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"]
Nov 03 19:51:59.985436 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:51:59.985374    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c95cd0_cf66_4a36_9c3e_f917ee445790.slice/crio-c3022ef8cab9bf20e3ebb931a37fb4b95e0158df49f49a880ea0144a33cf59f2 WatchSource:0}: Error finding container c3022ef8cab9bf20e3ebb931a37fb4b95e0158df49f49a880ea0144a33cf59f2: Status 404 returned error can't find the container with id c3022ef8cab9bf20e3ebb931a37fb4b95e0158df49f49a880ea0144a33cf59f2
Nov 03 19:51:59.994654 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.994608    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:59.994777 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.994706    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:51:59.994777 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:51:59.994751    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:00.036715 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:52:00.036679    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b7fd36_49dc_44d2_8b0b_284713245655.slice/crio-747127f7d6cf4b9dc4877c7f99c889a52a82992fc3fef6cb56dd51ef5ffa59cb WatchSource:0}: Error finding container 747127f7d6cf4b9dc4877c7f99c889a52a82992fc3fef6cb56dd51ef5ffa59cb: Status 404 returned error can't find the container with id 747127f7d6cf4b9dc4877c7f99c889a52a82992fc3fef6cb56dd51ef5ffa59cb
Nov 03 19:52:00.036853 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:00.036784    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"]
Nov 03 19:52:00.052895 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:00.052846    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:00.053007 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:00.052961    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:00.053082 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:00.053030    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:00.756099 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:00.756057    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:01.012172 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:01.010934    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" event={"ID":"e2b7fd36-49dc-44d2-8b0b-284713245655","Type":"ContainerStarted","Data":"1e63db2684ed19d8085bb80b751e0fec207dff5efc771072776158159b245b15"}
Nov 03 19:52:01.012172 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:01.011057    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" event={"ID":"e2b7fd36-49dc-44d2-8b0b-284713245655","Type":"ContainerStarted","Data":"747127f7d6cf4b9dc4877c7f99c889a52a82992fc3fef6cb56dd51ef5ffa59cb"}
Nov 03 19:52:01.015161 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:01.014383    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" event={"ID":"40c95cd0-cf66-4a36-9c3e-f917ee445790","Type":"ContainerStarted","Data":"9816c0426f3f554e6a5570384a178e4b4e503a4200e01dcc01469240c76b3a5f"}
Nov 03 19:52:01.015161 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:01.014437    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" event={"ID":"40c95cd0-cf66-4a36-9c3e-f917ee445790","Type":"ContainerStarted","Data":"c3022ef8cab9bf20e3ebb931a37fb4b95e0158df49f49a880ea0144a33cf59f2"}
Nov 03 19:52:01.047514 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:01.042206    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" podStartSLOduration=2.042189371 podStartE2EDuration="2.042189371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:01.037255774 +0000 UTC m=+873.192158194" watchObservedRunningTime="2026-11-03 19:52:01.042189371 +0000 UTC m=+873.197091792"
Nov 03 19:52:01.091473 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:01.091411    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" podStartSLOduration=2.091390275 podStartE2EDuration="2.091390275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:01.072988999 +0000 UTC m=+873.227891419" watchObservedRunningTime="2026-11-03 19:52:01.091390275 +0000 UTC m=+873.246292701"
Nov 03 19:52:01.558772 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:01.558737    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:01.681144 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:01.681113    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:52:01.681507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:01.681462    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:52:01.681653 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:01.681630    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:52:01.713520 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:01.710644    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:52:01.713520 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:01.710826    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:52:01.713520 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:01.710873    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:52:02.430274 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:02.430241    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:02.565450 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:02.565419    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:02.683080 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:02.682986    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:52:02.683361 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:02.683313    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:52:02.716613 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:02.716578    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:52:02.716892 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:02.716864    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:52:03.238630 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:03.238584    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:03.680565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:03.679924    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:52:03.680565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:03.680061    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:52:03.709399 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:03.709367    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:52:03.709593 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:03.709425    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:52:04.092539 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:04.091456    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:04.680506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:04.679886    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:52:04.680506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:04.679950    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:52:04.708748 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:04.708716    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:52:04.708965 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:04.708942    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:52:04.716657 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:04.716635    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:52:04.729934 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:04.729911    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:52:05.253773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:05.253739    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:05.439011 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:05.438980    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:52:05.866505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:05.863647    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:52:05.866505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:05.863709    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:52:05.965624 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:05.965592    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:52:05.968042 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:05.965825    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:52:06.422732 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:06.422650    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:06.447850 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:06.447806    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:06.686575 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:06.685596    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:52:06.686575 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:06.685668    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:52:06.713275 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:06.713236    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:52:06.713537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:06.713503    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:52:07.118053 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:07.118008    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:52:07.190606 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:07.190558    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:07.516509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:07.515990    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:07.520505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:07.516984    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:52:07.697521 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:07.692526    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:52:07.807231 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:07.807129    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:52:07.891842 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:07.891791    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:52:07.892158 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:07.892118    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:52:07.959266 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:07.959220    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Nov 03 19:52:08.140571 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.126919    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:08.140571 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.126991    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg" podUID="4d55596f-ce16-4eab-9c20-19ea62660132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:08.208509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.174683    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:52:08.208509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.204626    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:08.234506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.234439    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:52:08.262512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.234958    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:52:08.501507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.479290    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:08.563508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.537322    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:08.778203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.778111    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:52:08.778436 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.778409    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:52:08.778706 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.778678    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:52:08.782504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.779043    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:52:08.883257 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.883226    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:52:08.910858 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:08.910833    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:52:09.055985 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.055884    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:09.056607 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.056585    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:52:09.333763 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.333644    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:09.367096 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.354085    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:09.367734 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.354123    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:09.394531 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.368005    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:52:09.394531 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.368067    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:52:09.552557 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.505586    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:09.552557 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.506351    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:52:09.552557 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.546762    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:52:09.602070 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.601981    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:52:09.709602 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.709562    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:52:09.715515 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.709861    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:52:09.772993 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.772959    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:52:09.773301 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.773226    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:52:09.983136 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:09.983057    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:10.071444 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:10.069500    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:10.139063 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:10.135192    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:52:10.235718 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:10.235605    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:52:10.236323 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:10.236293    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:52:10.316508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:10.289601    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:52:10.369760 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:10.369718    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:10.473054 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:10.469680    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:10.622108 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:10.622012    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:10.710302 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:10.709811    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:52:10.710302 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:10.709876    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 19:52:10.710673 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:10.710649    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:52:10.710824 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:10.710785    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 19:52:11.125013 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:11.122361    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:52:11.147803 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:11.130132    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:11.228503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:11.212555    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:52:11.898013 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:11.897981    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:52:11.898632 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:11.898614    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:52:11.899302 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:11.899280    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 19:52:11.976592 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:11.976558    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:52:11.977304 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:11.977276    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:52:11.992539 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:11.988011    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 19:52:11.992539 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:11.988044    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:12.769510 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:12.768949    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:13.660876 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:13.660847    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:14.351097 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:14.351059    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:15.136826 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:15.136797    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:15.224041 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:52:15.224008    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:52:15.894806 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:15.894776    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:16.646656 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.646618    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:16.667381 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.667347    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"]
Nov 03 19:52:16.672251 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.672220    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.684812 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.684578    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d3d0df-openshift-default-dockercfg-m4lvp\""
Nov 03 19:52:16.693087 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.693060    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"]
Nov 03 19:52:16.813796 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.813764    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.813978 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.813878    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.813978 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.813910    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.814122 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.814101    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.814179 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.814140    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.814230 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.814182    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.814283 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.814230    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.814339 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.814307    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.814382 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.814340    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5w27\" (UniqueName: \"kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-kube-api-access-d5w27\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.915731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.915654    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.915731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.915698    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.915731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.915717    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5w27\" (UniqueName: \"kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-kube-api-access-d5w27\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.916077 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.915778    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.916499 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.916186    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.916499 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.916238    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.916499 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.916456    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.916663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.916519    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.916663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.916626    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.916663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.916648    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.916663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.916516    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.916842 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.916666    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.916969 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.916937    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.917084 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.916951    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.918791 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.918763    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.918976 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.918959    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.927900 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.927873    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5w27\" (UniqueName: \"kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-kube-api-access-d5w27\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:16.928247 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:16.928224    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:17.015345 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:17.015302    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:17.304006 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:17.303974    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"]
Nov 03 19:52:17.304893 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:52:17.304861    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129f5fbf_e7a2_436a_8a9f_1b52e42195f9.slice/crio-e9553064512ba82d31adfba800fa77d7edeb186e77605b20304b94ad9b432d64 WatchSource:0}: Error finding container e9553064512ba82d31adfba800fa77d7edeb186e77605b20304b94ad9b432d64: Status 404 returned error can't find the container with id e9553064512ba82d31adfba800fa77d7edeb186e77605b20304b94ad9b432d64
Nov 03 19:52:17.307390 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:17.307356    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:17.307459 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:17.307426    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:17.307525 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:17.307455    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:17.440124 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:17.440089    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:17.807244 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:17.807205    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" event={"ID":"129f5fbf-e7a2-436a-8a9f-1b52e42195f9","Type":"ContainerStarted","Data":"0306132eca4a1e60a4c3e87c7710961024013605bb30c4ff5f816c8fbcbce15b"}
Nov 03 19:52:17.807679 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:17.807266    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" event={"ID":"129f5fbf-e7a2-436a-8a9f-1b52e42195f9","Type":"ContainerStarted","Data":"e9553064512ba82d31adfba800fa77d7edeb186e77605b20304b94ad9b432d64"}
Nov 03 19:52:17.832403 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:17.832343    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podStartSLOduration=1.83232633 podStartE2EDuration="1.83232633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:17.830889322 +0000 UTC m=+889.985791753" watchObservedRunningTime="2026-11-03 19:52:17.83232633 +0000 UTC m=+889.987228750"
Nov 03 19:52:18.016020 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:18.015987    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:18.017366 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:18.017335    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:52:18.017505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:18.017396    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:52:18.277887 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:18.277693    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:19.016199 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:19.016160    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:52:19.016638 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:19.016228    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:52:19.366647 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:19.366447    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:20.024509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:20.023764    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:52:20.024509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:20.023841    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:52:21.018818 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:21.017906    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:52:21.018818 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:21.017971    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:52:21.282556 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:21.281670    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:21.851505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:21.850589    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.016508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:22.015888    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:52:22.016508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:22.015952    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:52:22.135546 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:22.121676    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.145222 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:22.144608    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.198517 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:22.191558    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.223505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:22.207595    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.223505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:22.212561    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.576916 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:22.546906    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:52:22.616629 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:22.613600    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.826551 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:22.813802    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:22.897571 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:22.893578    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.897571 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:22.894350    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:52:23.134723 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.134672    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:23.199735 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.199631    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:23.200358 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.200331    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:52:23.239633 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.239579    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:52:23.278518 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.212879    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:23.278518 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.200549    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:23.278518 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.267947    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:52:23.278518 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.276595    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:52:23.278518 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.276642    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:52:23.278518 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.276677    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:52:23.702507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.700615    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:52:23.702507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.702020    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:52:23.913421 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.913384    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:52:23.925507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:23.921660    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:52:24.047505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:24.047093    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:52:24.047505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:24.047169    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:52:24.252509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:24.204730    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:24.336584 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:24.206382    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:24.336584 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:24.215558    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:24.336584 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:24.335633    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:52:24.352980 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:24.352937    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:24.507668 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:24.507618    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:24.584663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:24.584613    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:24.776508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:24.731834    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:52:24.921509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:24.914594    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:24.925652 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:24.925609    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:52:25.016951 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.016906    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:52:25.017247 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.017218    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:52:25.141721 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.140591    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:52:25.521829 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.502903    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:52:25.531019 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.522644    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Nov 03 19:52:25.534998 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.522708    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 19:52:25.535467 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.522735    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:52:25.536951 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.528553    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:25.540017 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.531869    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:52:25.567644 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.567611    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:52:25.595507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.534695    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:25.822560 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.715278    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:52:25.897193 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.897159    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:52:25.909506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.728579    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 03 19:52:25.917541 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.759125    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:25.917724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.728627    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:25.938511 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.919924    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:52:25.938511 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.920292    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:52:25.938511 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.928551    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:25.974798 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:25.974770    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:52:26.054974 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:26.054252    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:52:26.055250 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:26.055219    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:52:26.084689 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:26.084585    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:26.132884 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:26.132849    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:52:26.218291 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:26.218258    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:52:26.245821 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:26.245795    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:52:26.369680 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:26.369593    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:52:26.390038 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:26.389075    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:52:26.390038 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:26.389975    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:52:26.399916 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:26.392812    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:52:26.554517 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:26.554466    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:27.015731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:27.015697    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:52:27.016133 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:27.015754    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:52:27.487024 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:27.486993    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:28.015947 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:28.015908    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:52:28.016379 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:28.015965    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 19:52:28.332361 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:28.332282    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:29.029729 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:29.029683    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:29.200358 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:29.200325    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:29.200808 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:29.200784    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:29.201631 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:29.201602    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 19:52:29.810220 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:29.810189    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:30.541327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:30.541288    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:31.408714 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:31.408680    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:32.135026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:32.134991    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:32.839614 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:32.839582    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:33.532452 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:33.532413    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:34.232746 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:34.232710    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:35.132803 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:35.132766    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:35.261683 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:35.261650    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:35.944864 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:35.944831    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:36.636299 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:36.636261    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:37.549651 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:37.549619    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:37.946197 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:37.946065    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:38.106140 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:38.106092    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:38.247508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:38.246766    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:39.250701 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:39.250599    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:39.330996 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:39.330947    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:39.340828 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:39.340787    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:39.402441 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:39.402387    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:52:39.403826 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:39.403799    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:39.806626 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:39.806577    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:39.928686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:39.928634    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:40.317783 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.308072    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:52:40.554603 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.499672    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:52:40.554603 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.506576    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:40.554603 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.507006    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:52:40.554603 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.507310    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:52:40.554603 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.511556    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:52:40.755506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.725669    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:52:40.755506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.725810    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:52:40.755506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.725893    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:52:40.755506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.725973    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:52:40.871506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.843848    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:40.871506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.844652    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:52:40.931377 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.931324    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:40.947934 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:40.947904    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:52:41.392506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:41.338131    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:41.410888 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:41.358823    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:41.571531 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:41.565917    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:41.576509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:41.571825    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:41.576509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:41.572111    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 19:52:41.576509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:41.572325    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:52:41.576509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:41.572560    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:52:41.576509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:41.572619    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:52:41.576509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:41.574768    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:52:41.856407 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:41.856358    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:42.033507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.010548    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:52:42.205932 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.205883    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:42.206276 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.206244    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:42.418504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.416675    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:52:42.418504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.417158    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Nov 03 19:52:42.418504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.417549    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:52:42.598554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.566096    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:52:42.598554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.583271    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:42.598554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.588051    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:52:42.598554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.588465    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:52:42.598554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.588735    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:42.882177 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.882082    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:42.953953 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.953601    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:42.953953 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:42.953677    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:43.054506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:43.012710    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:43.184823 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:43.184740    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:52:43.249174 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:43.249100    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:43.437742 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:43.437637    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:43.526052 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:43.526017    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:52:43.572617 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:43.572570    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:43.595469 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:43.595429    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:43.596036 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:43.596003    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:43.596825 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:43.596204    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:52:43.890516 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:43.890097    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:52:44.027514 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.026606    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:52:44.150739 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.147815    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:52:44.185616 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.185582    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:52:44.271971 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.271942    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:52:44.275399 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.275378    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:52:44.291722 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.291695    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:52:44.291959 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.291946    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:52:44.518805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.518767    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:44.663737 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.663704    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:44.840149 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.840055    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"]
Nov 03 19:52:44.844022 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.843989    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.868896 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.868855    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"]
Nov 03 19:52:44.880547 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.880514    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.880751 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.880562    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.880751 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.880710    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.880886 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.880789    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.880886 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.880845    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjtm8\" (UniqueName: \"kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-kube-api-access-qjtm8\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.880993 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.880924    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.880993 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.880955    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.880993 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.880978    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.881151 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.881052    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.933730 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.933692    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:44.982128 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.982086    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.982351 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.982190    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.982351 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.982254    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.982351 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.982332    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.982598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.982415    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.982598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.982530    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjtm8\" (UniqueName: \"kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-kube-api-access-qjtm8\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.982598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.982567    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.982773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.982745    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.982834 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.982777    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.982834 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.982816    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.982978 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.982946    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.983395 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.983342    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.983728 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.983706    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.983826 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.983807    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.985080 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.985056    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.985186 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.985154    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.997218 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.997106    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:44.997366 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:44.997255    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjtm8\" (UniqueName: \"kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-kube-api-access-qjtm8\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:45.170567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:45.169208    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:45.238657 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:52:45.238618    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:52:45.475978 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:52:45.475939    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb0655_e9f8_4d24_b524_5adbe643ba9d.slice/crio-2a27ca4f5454bc422fe51b2f500537f0f86b726ab7642e41e7aa8c5cd02f39cb WatchSource:0}: Error finding container 2a27ca4f5454bc422fe51b2f500537f0f86b726ab7642e41e7aa8c5cd02f39cb: Status 404 returned error can't find the container with id 2a27ca4f5454bc422fe51b2f500537f0f86b726ab7642e41e7aa8c5cd02f39cb
Nov 03 19:52:45.477083 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:45.477057    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"]
Nov 03 19:52:45.479224 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:45.479182    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:45.479318 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:45.479285    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:45.479377 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:45.479328    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:52:45.633909 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:45.633872    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" event={"ID":"b1cb0655-e9f8-4d24-b524-5adbe643ba9d","Type":"ContainerStarted","Data":"668fd6d24c0fb31ea276a6cfe69f0c2ee33384ca7e5d57a6b4aaab31948857db"}
Nov 03 19:52:45.634272 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:45.633942    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" event={"ID":"b1cb0655-e9f8-4d24-b524-5adbe643ba9d","Type":"ContainerStarted","Data":"2a27ca4f5454bc422fe51b2f500537f0f86b726ab7642e41e7aa8c5cd02f39cb"}
Nov 03 19:52:45.708812 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:45.708756    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" podStartSLOduration=1.708740495 podStartE2EDuration="1.708740495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:45.704406644 +0000 UTC m=+917.859309063" watchObservedRunningTime="2026-11-03 19:52:45.708740495 +0000 UTC m=+917.863642913"
Nov 03 19:52:45.869054 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:45.868965    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:45.993663 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:45.993625    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:46.163620 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:46.162295    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:46.169974 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:46.169945    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:46.171238 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:46.171208    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 19:52:46.171363 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:46.171262    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" podUID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 19:52:47.169989 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:47.169952    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 19:52:47.170652 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:47.170015    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" podUID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 19:52:47.381905 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:47.381852    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:48.171024 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:48.170990    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 19:52:48.171564 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:48.171052    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" podUID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 19:52:48.425706 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:48.425631    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:49.170237 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:49.170198    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 19:52:49.170413 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:49.170258    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" podUID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 19:52:49.422722 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:49.422637    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:50.170140 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:50.170092    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 19:52:50.170361 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:50.170161    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" podUID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 19:52:50.519673 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:50.519641    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:51.170517 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:51.170447    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 19:52:51.170709 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:51.170554    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" podUID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 19:52:51.694052 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:51.694012    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:52.170749 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:52.170710    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 19:52:52.170929 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:52.170774    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" podUID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 19:52:52.695735 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:52.695696    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:53.203036 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:53.202991    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:53.204391 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:53.204240    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:53.210359 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:53.208707    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 19:52:55.530538 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:55.529564    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:55.572509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:55.568560    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:55.609509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:55.608876    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:55.628129 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:55.617305    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 03 19:52:55.715626 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:55.693477    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:55.757427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:55.757389    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:56.101803 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:56.101751    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:56.277556 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:56.277526    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:56.436380 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:56.436272    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:56.597754 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:56.597699    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:56.598915 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:56.598801    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:52:56.730712 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:56.730679    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:56.913809 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:56.913771    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:56.960889 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:56.960860    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:52:59.858031 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:59.857996    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"]
Nov 03 19:52:59.863182 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:59.863154    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:52:59.871447 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:59.871416    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"]
Nov 03 19:52:59.912928 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:59.912887    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:52:59.913098 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:59.912950    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:52:59.913098 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:59.913024    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:52:59.913098 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:59.913052    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:52:59.913098 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:59.913089    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qchbk\" (UniqueName: \"kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-kube-api-access-qchbk\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:52:59.913303 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:59.913118    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:52:59.913364 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:59.913297    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:52:59.913364 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:59.913338    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:52:59.913468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:52:59.913443    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.014555 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.014511    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.014753 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.014632    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.014753 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.014707    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.014885 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.014786    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.014885 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.014856    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qchbk\" (UniqueName: \"kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-kube-api-access-qchbk\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.014997 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.014914    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.014997 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.014980    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.015104 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.015022    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.015104 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.015096    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.015199 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.015104    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.015277 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.015256    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.016230 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.016008    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.016230 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.016095    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.016397 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.016264    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.018081 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.018056    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.018319 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.018293    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.027329 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.027304    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.027735 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.027715    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qchbk\" (UniqueName: \"kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-kube-api-access-qchbk\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.180393 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.180308    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:00.469685 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.469649    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"]
Nov 03 19:53:00.486108 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.486070    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:00.486204 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.486140    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:00.486204 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.486169    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:00.661018 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.660963    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:00.804112 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.804031    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:00.934742 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:00.934706    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:01.079646 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.079548    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2" event={"ID":"fdde0ea7-99e2-4098-8e03-db9b8533aab9","Type":"ContainerStarted","Data":"a6c5fdf52913cd2217a9fac969383336ded73621f32982049e55134458f8cc6a"}
Nov 03 19:53:01.079646 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.079620    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2" event={"ID":"fdde0ea7-99e2-4098-8e03-db9b8533aab9","Type":"ContainerStarted","Data":"f88c24c7100bc819bd63268788cbd82ddd1107c1fa142d4d40306eb3253ca486"}
Nov 03 19:53:01.125043 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.124991    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2" podStartSLOduration=2.124975068 podStartE2EDuration="2.124975068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:01.122270811 +0000 UTC m=+933.277173239" watchObservedRunningTime="2026-11-03 19:53:01.124975068 +0000 UTC m=+933.279877488"
Nov 03 19:53:01.182123 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.182046    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:01.182696 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.182658    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:53:01.182868 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.182849    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2" podUID="fdde0ea7-99e2-4098-8e03-db9b8533aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:53:01.786419 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.786384    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"]
Nov 03 19:53:01.800004 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.799966    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"]
Nov 03 19:53:01.800191 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.800101    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:01.969711 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.969653    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:01.971439 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.971120    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:01.971680 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.971662    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:01.971854 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.971836    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:01.972033 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.972017    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:01.972189 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.972174    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:01.972409 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.972390    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:01.972597 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.972579    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:01.972793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.972777    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldlqg\" (UniqueName: \"kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-kube-api-access-ldlqg\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:01.973468 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:01.973406    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.074753 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.074623    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldlqg\" (UniqueName: \"kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-kube-api-access-ldlqg\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.074753 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.074688    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.074982 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.074815    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.074982 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.074852    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.074982 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.074877    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.074982 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.074948    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.074982 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.074975    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.075256 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.075105    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.075256 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.075142    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.076501 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.076452    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.077259 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.077213    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.077529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.077507    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.079600 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.079571    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.083592 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.083566    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.088772 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.088744    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.088884 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.088822    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.093746 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.093715    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldlqg\" (UniqueName: \"kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-kube-api-access-ldlqg\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.094722 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.094695    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.114114 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.114081    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:02.118557 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.117679    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:02.181794 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.181763    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:53:02.181962 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.181824    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2" podUID="fdde0ea7-99e2-4098-8e03-db9b8533aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:53:02.290746 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.252917    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:02.527442 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.527409    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"]
Nov 03 19:53:02.530766 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:53:02.530735    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee48fa2_b463_4d56_b9e8_ec887c95ba29.slice/crio-478146c9833ecc78adc251d8a5a153a983b685be477d477414e06029903b4d01 WatchSource:0}: Error finding container 478146c9833ecc78adc251d8a5a153a983b685be477d477414e06029903b4d01: Status 404 returned error can't find the container with id 478146c9833ecc78adc251d8a5a153a983b685be477d477414e06029903b4d01
Nov 03 19:53:02.565364 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.565317    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:02.565571 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.565401    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:02.565571 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:02.565437    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:03.106602 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:03.106567    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" event={"ID":"bee48fa2-b463-4d56-b9e8-ec887c95ba29","Type":"ContainerStarted","Data":"661b82ffaab9c686dac16c5fef008e78bfc13c0a098d8459b878c41e486d5af8"}
Nov 03 19:53:03.107171 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:03.107146    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" event={"ID":"bee48fa2-b463-4d56-b9e8-ec887c95ba29","Type":"ContainerStarted","Data":"478146c9833ecc78adc251d8a5a153a983b685be477d477414e06029903b4d01"}
Nov 03 19:53:03.119059 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:03.119029    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:03.120189 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:03.120161    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:53:03.120309 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:03.120213    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" podUID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:53:03.210580 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:03.207568    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" podStartSLOduration=2.207547691 podStartE2EDuration="2.207547691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:03.131090471 +0000 UTC m=+935.285992907" watchObservedRunningTime="2026-11-03 19:53:03.207547691 +0000 UTC m=+935.362450104"
Nov 03 19:53:03.211564 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:03.211533    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:53:03.211785 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:03.211758    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2" podUID="fdde0ea7-99e2-4098-8e03-db9b8533aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:53:03.221448 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:03.221404    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:03.960159 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:03.960099    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:04.106495 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:04.106186    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:04.119142 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:04.119107    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:53:04.119568 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:04.119174    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" podUID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:53:04.196508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:04.182145    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:53:04.196508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:04.182219    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2" podUID="fdde0ea7-99e2-4098-8e03-db9b8533aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:53:04.998506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:04.997823    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:05.119858 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:05.119823    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:53:05.120355 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:05.119887    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" podUID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:53:05.192638 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:05.192603    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:53:05.192914 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:05.192882    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2" podUID="fdde0ea7-99e2-4098-8e03-db9b8533aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:53:05.880505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:05.879380    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:06.118339 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.118296    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:53:06.118543 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.118372    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" podUID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:53:06.186561 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.185706    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:53:06.186561 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.185825    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2" podUID="fdde0ea7-99e2-4098-8e03-db9b8533aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:53:06.517504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.516809    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"]
Nov 03 19:53:06.527068 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.522086    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:06.527068 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.526090    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d8d238bf-serving-cert\""
Nov 03 19:53:06.529134 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.529109    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"]
Nov 03 19:53:06.652724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.624756    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:06.652724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.631887    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-metrics-certs\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:06.652724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.632013    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8cmw\" (UniqueName: \"kubernetes.io/projected/9d5154a2-122f-4b75-af12-b023995483e2-kube-api-access-h8cmw\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:06.652724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.632075    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-maas-api-tls\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:06.652724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.632111    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5154a2-122f-4b75-af12-b023995483e2-openshift-service-ca\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:06.749508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.740250    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8cmw\" (UniqueName: \"kubernetes.io/projected/9d5154a2-122f-4b75-af12-b023995483e2-kube-api-access-h8cmw\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:06.749508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.740318    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-maas-api-tls\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:06.749508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.740357    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5154a2-122f-4b75-af12-b023995483e2-openshift-service-ca\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:06.749508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.740553    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-metrics-certs\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:06.773038 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.758711    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5154a2-122f-4b75-af12-b023995483e2-openshift-service-ca\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:06.773038 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:53:06.764228    5659 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-serving-cert: secret "maas-api-e2e-mt-d8d238bf-serving-cert" not found
Nov 03 19:53:06.773038 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:53:06.764299    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-maas-api-tls podName:9d5154a2-122f-4b75-af12-b023995483e2 nodeName:}" failed. No retries permitted until 2026-11-03 19:53:07.264277653 +0000 UTC m=+939.419180068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-maas-api-tls") pod "maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd" (UID: "9d5154a2-122f-4b75-af12-b023995483e2") : secret "maas-api-e2e-mt-d8d238bf-serving-cert" not found
Nov 03 19:53:06.773038 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.765104    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:06.787971 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.787934    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-metrics-certs\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:06.789530 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.789460    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8cmw\" (UniqueName: \"kubernetes.io/projected/9d5154a2-122f-4b75-af12-b023995483e2-kube-api-access-h8cmw\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:06.903291 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:06.902214    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:07.122466 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:07.122374    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:53:07.122773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:07.122740    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" podUID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:53:07.190072 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:07.190038    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:53:07.212982 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:07.212847    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2" podUID="fdde0ea7-99e2-4098-8e03-db9b8533aab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 19:53:07.280508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:07.280439    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-maas-api-tls\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:07.304225 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:07.304161    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-maas-api-tls\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:07.472624 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:07.471875    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:07.567774 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:07.567732    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:08.031375 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:08.031341    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"]
Nov 03 19:53:08.062549 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:08.052553    5659 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:53:08.118320 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:08.118283    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:53:08.118642 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:08.118610    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" podUID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:53:08.237777 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:08.237737    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd" event={"ID":"9d5154a2-122f-4b75-af12-b023995483e2","Type":"ContainerStarted","Data":"13638a0057d50f7d57d1818313f2ac7a476020365b789a393d6502855326477f"}
Nov 03 19:53:08.361567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:08.360959    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:08.850508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:08.848812    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:08.850508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:08.850257    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:08.858537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:08.858477    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 19:53:09.118683 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:09.118640    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 19:53:09.118869 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:09.118715    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" podUID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 19:53:09.235317 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:09.235279    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:09.261517 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:09.256867    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd" event={"ID":"9d5154a2-122f-4b75-af12-b023995483e2","Type":"ContainerStarted","Data":"3b9416d3f5891d26e8529f293459eb7e4bebf0028680020026e34e426aa86cd6"}
Nov 03 19:53:09.261517 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:09.256966    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:10.025109 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:10.025077    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:10.348115 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:10.348035    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:10.349286 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:10.349264    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:10.350167 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:10.350145    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 19:53:10.378668 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:10.378607    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd" podStartSLOduration=3.736812006 podStartE2EDuration="4.378587585s" totalImagesPullingTime="641.775579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:09.280829055 +0000 UTC m=+941.435731490" watchObservedRunningTime="2026-11-03 19:53:10.378587585 +0000 UTC m=+942.533490006"
Nov 03 19:53:10.845910 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:10.845879    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:11.641533 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:11.641462    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:12.345667 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:12.345633    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:12.962334 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:12.962279    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:13.113685 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:13.113648    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:14.018621 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:14.018577    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:53:14.194508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:14.193621    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:14.409808 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:14.409711    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:14.492623 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:14.492574    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Nov 03 19:53:14.568620 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:14.568571    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:14.604717 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:14.604657    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:14.971506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:14.968573    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.194679 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.194624    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.196080 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.196055    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:53:15.220933 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.220709    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:15.410613 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.410513    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.454135 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.411358    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:53:15.505517 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.418825    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:53:15.594860 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.594802    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.619656 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.605251    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:53:15.650104 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.650061    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:53:15.678563 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.677592    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.678563 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.677695    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 19:53:15.678563 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.678143    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:53:15.804519 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.717814    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.804519 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.718783    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 03 19:53:15.804519 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:53:15.766973    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:53:15.883615 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.872336    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:15.922525 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.910188    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.992542 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:15.976579    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:16.042508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:16.030584    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:53:16.102197 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:16.101023    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:53:16.102197 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:16.101094    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:16.104921 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:16.102579    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:16.233628 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:16.232126    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:16.410173 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:16.393026    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:53:16.582088 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:16.458538    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:53:16.663022 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:16.661628    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:16.693747 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:16.693697    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:53:16.706692 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:16.706138    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:53:16.762263 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:16.757569    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:17.004557 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:16.906222    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:53:17.025507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:16.957813    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:17.342570 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.323951    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:17.426739 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.396613    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Nov 03 19:53:17.493537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.481580    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:53:17.523268 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.523233    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:53:17.523597 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.523552    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:17.530064 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.529879    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:53:17.530064 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.529934    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"]
Nov 03 19:53:17.591538 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.536546    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:17.591538 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.578451    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:53:17.603625 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.601417    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:17.666819 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.666614    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:17.666819 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.666701    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:53:17.735791 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.735753    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:53:17.736056 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.736039    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 19:53:17.769505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.753806    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:53:17.770166 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.736252    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:17.799861 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.799825    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:17.957506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.954038    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:17.961839 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.961799    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:17.970372 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.970339    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26jjx\" (UniqueName: \"kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-kube-api-access-26jjx\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:17.970680 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.970657    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:17.986731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.978958    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:17.986731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.979008    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:17.986731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.979053    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:17.986731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.979198    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:17.986731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:17.979290    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.034921 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.034866    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:18.127504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.106525    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.133544 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.130371    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26jjx\" (UniqueName: \"kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-kube-api-access-26jjx\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.133544 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.130520    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.133544 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.130613    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.133544 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.130651    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.133544 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.130699    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.133544 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.130857    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.133544 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.130945    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.133544 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.131090    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.172596 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.162938    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.172596 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.163185    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.172596 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.164114    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.202099 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.202061    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.202645 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.202617    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.237512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.229666    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.261937 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.261898    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 19:53:18.307506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.307384    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26jjx\" (UniqueName: \"kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-kube-api-access-26jjx\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.307506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.307466    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"]
Nov 03 19:53:18.331505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.308056    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.332125 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.332093    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.430098 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.429775    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:18.503532 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.482905    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 19:53:18.558831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.529579    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:18.581922 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.532229    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 19:53:18.582538 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.582476    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:18.596621 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.596566    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:18.676185 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.602726    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:18.676185 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.671084    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:18.811245 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.810567    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:18.893269 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.892819    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g44hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:53:18.893269 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.892880    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx" podUID="a16a0bd3-5dd9-496b-9708-ee0c4091090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:18.895738 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.895714    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 19:53:18.944689 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.943543    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 19:53:18.955029 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:18.955002    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 19:53:19.048574 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:19.047939    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 19:53:19.079733 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:19.079707    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 19:53:19.339231 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:19.339190    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"]
Nov 03 19:53:19.410886 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:19.410843    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:19.411451 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:19.411425    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:19.411636 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:19.411617    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:53:19.484364 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:19.483658    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:19.509217 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:19.507803    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 19:53:19.527528 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:19.526738    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 19:53:19.708940 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:19.703227    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:19.806125 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:19.796942    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p" event={"ID":"85068d81-d023-4e25-a6c1-3aff06dc9bd7","Type":"ContainerStarted","Data":"85f4dd7ed00c14d6a870c6ad4c97fc5ebcdb2fee8bb99cc45e876ac7f27a13c0"}
Nov 03 19:53:20.805018 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:20.804984    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p" event={"ID":"85068d81-d023-4e25-a6c1-3aff06dc9bd7","Type":"ContainerStarted","Data":"340408e2cfad879dd92020952cad19999f5f5c2b584427dc487ae6c66b386679"}
Nov 03 19:53:20.831828 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:20.831766    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p" podStartSLOduration=4.831748201 podStartE2EDuration="4.831748201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:20.828845049 +0000 UTC m=+952.983747469" watchObservedRunningTime="2026-11-03 19:53:20.831748201 +0000 UTC m=+952.986650674"
Nov 03 19:53:21.062564 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:21.055555    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:21.430960 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:21.430884    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:21.440507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:21.432792    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:53:21.440507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:21.432854    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p" podUID="85068d81-d023-4e25-a6c1-3aff06dc9bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:53:21.852966 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:21.852930    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:22.430999 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:22.430964    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:53:22.431195 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:22.431029    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p" podUID="85068d81-d023-4e25-a6c1-3aff06dc9bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:53:22.853516 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:22.852985    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:23.431596 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:23.431559    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:53:23.431894 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:23.431865    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p" podUID="85068d81-d023-4e25-a6c1-3aff06dc9bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:53:23.761013 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:23.760963    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:23.892803 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:23.892756    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:24.174222 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:24.174090    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:24.310425 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:24.310384    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:24.435672 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:24.435587    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:53:24.435914 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:24.435886    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p" podUID="85068d81-d023-4e25-a6c1-3aff06dc9bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:53:24.953537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:24.946135    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:25.431899 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:25.431862    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:53:25.432194 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:25.432166    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p" podUID="85068d81-d023-4e25-a6c1-3aff06dc9bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:53:26.092535 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:26.092499    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:26.431801 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:26.431700    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:53:26.431801 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:26.431786    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p" podUID="85068d81-d023-4e25-a6c1-3aff06dc9bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 19:53:27.001868 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:27.001832    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:27.436374 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:27.436287    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:27.436857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:27.436728    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:27.437681 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:27.437655    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 19:53:28.044749 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:28.044720    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:29.591732 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:29.590609    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 19:53:29.591732 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:29.590884    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:53:29.632513 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:29.622574    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 19:53:29.632513 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:29.631651    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:29.700509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:29.697962    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:29.845702 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:29.845565    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:29.879801 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:29.879768    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:30.833611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:30.833531    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:31.410016 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:31.409979    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:32.338831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:32.338798    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:33.054111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.054075    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:33.329707 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.329614    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"]
Nov 03 19:53:33.335154 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.335133    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.345800 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.345776    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d8d238bf\"/\"kube-root-ca.crt\""
Nov 03 19:53:33.346690 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.346670    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d8d238bf\"/\"e2e-lifecycle-model-d8d238bf-kserve-self-signed-certs\""
Nov 03 19:53:33.346974 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.346940    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d8d238bf\"/\"openshift-service-ca.crt\""
Nov 03 19:53:33.346974 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.346141    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d8d238bf\"/\"default-dockercfg-xxprv\""
Nov 03 19:53:33.361671 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.361649    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"]
Nov 03 19:53:33.398174 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.398133    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:33.427256 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.427222    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-dshm\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.427437 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.427352    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-home\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.427437 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.427385    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxzmw\" (UniqueName: \"kubernetes.io/projected/243a0d94-99af-4a65-b019-64f05ebe95a5-kube-api-access-dxzmw\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.427563 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.427538    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243a0d94-99af-4a65-b019-64f05ebe95a5-tls-certs\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.427724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.427708    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-model-cache\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.427768 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.427751    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-tmp-dir\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.530326 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.530283    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-tmp-dir\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.530615 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.530593    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-dshm\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.546507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.530766    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-home\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.546799 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.546768    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxzmw\" (UniqueName: \"kubernetes.io/projected/243a0d94-99af-4a65-b019-64f05ebe95a5-kube-api-access-dxzmw\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.547003 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.546985    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243a0d94-99af-4a65-b019-64f05ebe95a5-tls-certs\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.547352 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.547322    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-model-cache\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.548064 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.537806    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-tmp-dir\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.548522 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.548475    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-model-cache\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.548664 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.548499    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-home\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.559673 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.558532    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-dshm\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.580298 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.580166    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243a0d94-99af-4a65-b019-64f05ebe95a5-tls-certs\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.613083 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.613049    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxzmw\" (UniqueName: \"kubernetes.io/projected/243a0d94-99af-4a65-b019-64f05ebe95a5-kube-api-access-dxzmw\") pod \"e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") " pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.653222 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.653184    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:33.969096 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:53:33.969067    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod243a0d94_99af_4a65_b019_64f05ebe95a5.slice/crio-8a4acc9948316f56b855489b56365f3787a65e44f5149e10bc7f9789d6a219d8 WatchSource:0}: Error finding container 8a4acc9948316f56b855489b56365f3787a65e44f5149e10bc7f9789d6a219d8: Status 404 returned error can't find the container with id 8a4acc9948316f56b855489b56365f3787a65e44f5149e10bc7f9789d6a219d8
Nov 03 19:53:33.969437 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:33.969416    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"]
Nov 03 19:53:34.049369 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:34.049334    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:34.839542 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:34.839511    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:34.969215 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:34.969181    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8" event={"ID":"243a0d94-99af-4a65-b019-64f05ebe95a5","Type":"ContainerStarted","Data":"a2fd9ee51d2c29ec9431e12ece3d983f507434f0c9d449b55b86f4770aa55ec3"}
Nov 03 19:53:34.969215 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:34.969219    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8" event={"ID":"243a0d94-99af-4a65-b019-64f05ebe95a5","Type":"ContainerStarted","Data":"8a4acc9948316f56b855489b56365f3787a65e44f5149e10bc7f9789d6a219d8"}
Nov 03 19:53:34.992634 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:34.992582    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8" podStartSLOduration=1.992569263 podStartE2EDuration="1.992569263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:34.989689924 +0000 UTC m=+967.144592343" watchObservedRunningTime="2026-11-03 19:53:34.992569263 +0000 UTC m=+967.147471682"
Nov 03 19:53:35.359660 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:35.359632    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:35.653398 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:35.653321    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:35.732597 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:35.732556    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:36.456596 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:36.456566    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:37.143794 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:37.143761    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:37.845876 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:37.845846    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:38.635474 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:38.635438    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:39.432539 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:39.432497    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:40.243558 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:40.243521    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:40.942819 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:40.942772    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:41.828285 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:41.828226    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:42.641423 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:42.641391    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:42.983353 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:42.983307    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:43.371209 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:43.371128    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:43.538519 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:43.533880    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:43.653842 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:43.653742    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:43.662823 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:43.662765    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:43.668153 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:43.668126    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:44.045693 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:44.045661    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:53:44.257716 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:44.257651    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:45.044125 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:45.044097    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:45.216055 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:45.215992    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:45.218286 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:53:45.218255    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:53:45.663027 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:45.662995    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:45.790107 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:45.790077    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:45.920707 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:45.920631    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:46.723786 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:46.723755    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:47.186729 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:47.186647    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:47.334299 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:47.334258    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:47.589664 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:47.587533    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:47.717284 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:47.717253    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:48.070754 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:48.070701    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:48.497945 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:48.497911    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:48.645915 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:48.645883    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:49.511178 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:49.511146    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:51.147428 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:51.147324    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:51.984280 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:51.984164    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:52.082635 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:52.082588    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8" podUID="243a0d94-99af-4a65-b019-64f05ebe95a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:52.148524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:52.148466    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:52.548514 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:52.548464    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:53.731785 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:53.731751    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:53.807166 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:53.807121    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"]
Nov 03 19:53:53.812512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:53.812470    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"
Nov 03 19:53:53.818168 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:53.818121    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0a2237\""
Nov 03 19:53:53.826725 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:53.818878    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"]
Nov 03 19:53:53.826725 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:53.819140    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0a2237-dockercfg-s7746\""
Nov 03 19:53:53.868015 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:53.867960    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:53.887495 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:53.887452    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzckh\" (UniqueName: \"kubernetes.io/projected/4547bb97-69e8-4103-a263-3284b988223b-kube-api-access-xzckh\") pod \"payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7\" (UID: \"4547bb97-69e8-4103-a263-3284b988223b\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"
Nov 03 19:53:53.887700 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:53.887556    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4547bb97-69e8-4103-a263-3284b988223b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7\" (UID: \"4547bb97-69e8-4103-a263-3284b988223b\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"
Nov 03 19:53:53.998633 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:53.998536    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4547bb97-69e8-4103-a263-3284b988223b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7\" (UID: \"4547bb97-69e8-4103-a263-3284b988223b\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"
Nov 03 19:53:53.999156 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:53.999129    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzckh\" (UniqueName: \"kubernetes.io/projected/4547bb97-69e8-4103-a263-3284b988223b-kube-api-access-xzckh\") pod \"payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7\" (UID: \"4547bb97-69e8-4103-a263-3284b988223b\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"
Nov 03 19:53:54.003451 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:54.000330    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4547bb97-69e8-4103-a263-3284b988223b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7\" (UID: \"4547bb97-69e8-4103-a263-3284b988223b\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"
Nov 03 19:53:54.022225 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:54.022173    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzckh\" (UniqueName: \"kubernetes.io/projected/4547bb97-69e8-4103-a263-3284b988223b-kube-api-access-xzckh\") pod \"payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7\" (UID: \"4547bb97-69e8-4103-a263-3284b988223b\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"
Nov 03 19:53:54.136035 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:54.135993    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:54.149763 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:54.149732    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"
Nov 03 19:53:54.475765 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:53:54.475725    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4547bb97_69e8_4103_a263_3284b988223b.slice/crio-ee8ff3095f18cc9c997033cad4e653574cf135cf5ba5ef5aa5e450408945d284 WatchSource:0}: Error finding container ee8ff3095f18cc9c997033cad4e653574cf135cf5ba5ef5aa5e450408945d284: Status 404 returned error can't find the container with id ee8ff3095f18cc9c997033cad4e653574cf135cf5ba5ef5aa5e450408945d284
Nov 03 19:53:54.477138 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:54.477109    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"]
Nov 03 19:53:55.168120 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:55.168074    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7" event={"ID":"4547bb97-69e8-4103-a263-3284b988223b","Type":"ContainerStarted","Data":"ee8ff3095f18cc9c997033cad4e653574cf135cf5ba5ef5aa5e450408945d284"}
Nov 03 19:53:55.255115 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:55.255062    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:56.173678 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:56.169094    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:56.302839 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:56.302805    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:56.446571 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:56.443478    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:57.235991 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:57.235959    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:58.038990 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:58.038959    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:58.202458 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:58.202422    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7" event={"ID":"4547bb97-69e8-4103-a263-3284b988223b","Type":"ContainerStarted","Data":"440e3e121f429f9a8302e1c55378e8cb8f7af41bb679ff488e3e570d9bcbd35d"}
Nov 03 19:53:58.202667 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:58.202615    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"
Nov 03 19:53:58.225372 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:58.225311    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7" podStartSLOduration=2.516975359 podStartE2EDuration="5.225291253s" totalImagesPullingTime="2.708315894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:58.219959286 +0000 UTC m=+990.374861708" watchObservedRunningTime="2026-11-03 19:53:58.225291253 +0000 UTC m=+990.380193676"
Nov 03 19:53:58.829102 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:58.829071    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:59.660163 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:53:59.660124    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:00.244460 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.244430    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"]
Nov 03 19:54:00.252187 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.252136    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.261733 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.261705    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0a2237-serving-cert\""
Nov 03 19:54:00.273524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.269497    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"]
Nov 03 19:54:00.339712 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.339673    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87abccc8-39bc-412e-b61e-68a960e5d269-openshift-service-ca\") pod \"maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.339891 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.339755    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqql\" (UniqueName: \"kubernetes.io/projected/87abccc8-39bc-412e-b61e-68a960e5d269-kube-api-access-nlqql\") pod \"maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.339891 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.339815    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-maas-api-tls\") pod \"maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.340070 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.340045    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-metrics-certs\") pod \"maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.341825 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.341792    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:00.451537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.451194    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-metrics-certs\") pod \"maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.451537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.451376    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87abccc8-39bc-412e-b61e-68a960e5d269-openshift-service-ca\") pod \"maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.451537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.451427    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqql\" (UniqueName: \"kubernetes.io/projected/87abccc8-39bc-412e-b61e-68a960e5d269-kube-api-access-nlqql\") pod \"maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.451537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.451499    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-maas-api-tls\") pod \"maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.453428 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.452791    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87abccc8-39bc-412e-b61e-68a960e5d269-openshift-service-ca\") pod \"maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.457930 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.457901    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-maas-api-tls\") pod \"maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.462103 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.462072    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-metrics-certs\") pod \"maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.475616 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.473849    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqql\" (UniqueName: \"kubernetes.io/projected/87abccc8-39bc-412e-b61e-68a960e5d269-kube-api-access-nlqql\") pod \"maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.481172 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.481146    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:00.586338 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.586248    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:00.873518 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:00.873335    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"]
Nov 03 19:54:01.223078 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:01.223037    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx" event={"ID":"87abccc8-39bc-412e-b61e-68a960e5d269","Type":"ContainerStarted","Data":"28353adbfc252ab474d2a0f88586902004f3502ff0bd5cbe5d88dc447bac02f2"}
Nov 03 19:54:01.436628 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:01.436599    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:02.231590 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:02.231553    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx" event={"ID":"87abccc8-39bc-412e-b61e-68a960e5d269","Type":"ContainerStarted","Data":"eacf4383b5dd4280fb9e2fd7696d20c7beeaf0bc13777cb3ef7250c42f5206ff"}
Nov 03 19:54:02.231822 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:02.231618    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:02.232123 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:02.232102    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:02.259939 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:02.259473    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx" podStartSLOduration=1.7735881770000002 podStartE2EDuration="2.259451606s" totalImagesPullingTime="485.863429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:02.257249995 +0000 UTC m=+994.412152420" watchObservedRunningTime="2026-11-03 19:54:02.259451606 +0000 UTC m=+994.414354024"
Nov 03 19:54:02.978700 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:02.978670    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:03.657692 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:03.657659    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:04.442757 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:04.442710    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:05.217839 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:05.217805    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:05.847884 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:05.847840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:06.530312 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:06.530257    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:07.351789 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:07.351759    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:08.058773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:08.058743    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:08.243743 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:08.243713    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:08.751071 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:08.751034    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:09.212811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:09.212784    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"
Nov 03 19:54:09.415716 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:09.415684    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:10.058624 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:10.058588    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:11.011623 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:11.011594    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:11.734764 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:11.734727    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:12.491530 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:12.491498    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:12.830931 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:12.830857    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:13.154126 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:13.154035    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:14.217235 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:14.217193    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:14.975985 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:14.975949    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:15.219626 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:54:15.219586    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:54:16.053237 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:16.053203    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:16.819412 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:16.819371    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:16.955318 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:16.955286    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:17.450824 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:17.450782    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:17.751466 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:17.751429    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:17.907072 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:17.907023    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:18.042797 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.042713    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:18.827753 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.827718    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"]
Nov 03 19:54:18.833412 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.833385    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:18.840453 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.840429    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 19:54:18.840679 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.840500    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 19:54:18.840815 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.840530    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 19:54:18.843572 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.843169    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nh6bx\""
Nov 03 19:54:18.848196 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.846546    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"]
Nov 03 19:54:18.899939 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.897202    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:18.906641 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.906606    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:18.907074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.907036    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:18.907187 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.907105    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:18.907671 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.907302    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:18.907671 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.907537    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t45f4\" (UniqueName: \"kubernetes.io/projected/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-kube-api-access-t45f4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:18.907798 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.907675    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:18.907798 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:18.907750    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.009036 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.008999    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.009226 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.009090    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.009226 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.009133    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.009226 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.009197    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.009409 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.009260    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t45f4\" (UniqueName: \"kubernetes.io/projected/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-kube-api-access-t45f4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.009409 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.009354    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.009409 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.009390    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.010025 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.010000    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.011846 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.011817    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.012091 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.012068    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.012177 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.012124    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.012245 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.012231    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.021838 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.021813    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.022047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.022032    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t45f4\" (UniqueName: \"kubernetes.io/projected/fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a-kube-api-access-t45f4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8qktw\" (UID: \"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.159372 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.158988    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:19.480753 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.480725    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"]
Nov 03 19:54:19.483209 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:54:19.483161    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6d8d2d_637f_4f1a_9834_dffb9dbc2c6a.slice/crio-b3eb937c0ec3cfcf751771cf11e0a437eb6e444890848f0d3ee01ccc5071db75 WatchSource:0}: Error finding container b3eb937c0ec3cfcf751771cf11e0a437eb6e444890848f0d3ee01ccc5071db75: Status 404 returned error can't find the container with id b3eb937c0ec3cfcf751771cf11e0a437eb6e444890848f0d3ee01ccc5071db75
Nov 03 19:54:19.756939 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:19.756840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:20.399424 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:20.399387    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw" event={"ID":"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a","Type":"ContainerStarted","Data":"b3eb937c0ec3cfcf751771cf11e0a437eb6e444890848f0d3ee01ccc5071db75"}
Nov 03 19:54:20.760659 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:20.756207    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:21.370517 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:21.365704    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:21.501811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:21.501781    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:21.748530 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:21.746818    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:22.539758 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:22.539721    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:22.616822 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:22.616786    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:22.616917 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:22.616853    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:22.699148 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:22.699119    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:23.425942 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:23.425904    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw" event={"ID":"fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a","Type":"ContainerStarted","Data":"04be770b76e578d582c7f0555fdcac45d7808d6909f7fcf88ef7d844dac33a23"}
Nov 03 19:54:23.426335 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:23.426309    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:23.428889 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:23.428861    5659 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8qktw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:54:23.429002 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:23.428912    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw" podUID="fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:54:23.459178 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:23.459107    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw" podStartSLOduration=2.3292450430000002 podStartE2EDuration="5.459083978s" totalImagesPullingTime="3.129838935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:23.454875559 +0000 UTC m=+1015.609777978" watchObservedRunningTime="2026-11-03 19:54:23.459083978 +0000 UTC m=+1015.613986403"
Nov 03 19:54:23.541507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:23.541422    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:23.711510 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:23.709167    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:23.871504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:23.871244    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:24.434372 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:24.434337    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8qktw"
Nov 03 19:54:24.496819 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:24.494056    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:25.358271 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:25.356226    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:26.528352 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:26.528317    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:27.391263 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:27.391223    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:27.526401 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:27.526370    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:28.284278 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:28.284239    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:28.734288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:28.734254    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:28.864053 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:28.864017    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:29.544031 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:29.543987    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:29.690254 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:29.690215    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:30.671896 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:30.671861    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:31.553439 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:31.553388    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:32.345118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:32.345069    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:33.141144 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:33.141112    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:33.818218 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:33.818185    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:34.454790 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:34.454757    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:35.217337 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:35.217303    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:35.348414 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:35.348332    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:35.482759 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:35.482672    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:36.271979 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:36.271925    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:37.045423 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:37.045391    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:37.516920 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:37.516888    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:37.872028 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:37.871903    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:38.598233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:38.598203    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:39.616235 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:39.616206    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:40.382594 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:40.382564    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:41.042028 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:41.041994    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:41.443981 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:41.443886    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:41.819015 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:41.818964    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:42.474191 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:42.474158    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:43.020034 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:43.020000    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:43.236912 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:43.236878    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:44.043754 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:44.043690    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:44.669346 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:44.669189    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:45.027809 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:45.027768    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:45.217582 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:54:45.217546    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:54:45.842555 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:45.842464    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:46.740158 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:46.740124    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:47.027494 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:47.027376    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"]
Nov 03 19:54:47.030787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:47.030763    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd9ceca-m98f4"]
Nov 03 19:54:47.473528 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:47.473498    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:48.013507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:48.012407    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf" path="/var/lib/kubelet/pods/bf5251ea-f47a-4b1c-88ce-39a14a4cc8bf/volumes"
Nov 03 19:54:48.286522 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:48.286412    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"]
Nov 03 19:54:48.440515 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:48.440023    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:48.506514 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:48.484152    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:48.602735 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:48.587570    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:48.614814 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:48.614780    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:48.775047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:48.775005    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:48.919388 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:48.919285    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:48.919822 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:48.919597    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g44hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:54:48.920072 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:48.920049    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx" podUID="a16a0bd3-5dd9-496b-9708-ee0c4091090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:49.691750 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:49.691702    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:50.477062 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:50.477018    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:51.120214 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:51.120182    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:51.929242 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:51.929191    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:52.479184 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:52.479149    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"]
Nov 03 19:54:52.479983 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:52.479950    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx" podUID="87abccc8-39bc-412e-b61e-68a960e5d269" containerName="maas-api" containerID="cri-o://eacf4383b5dd4280fb9e2fd7696d20c7beeaf0bc13777cb3ef7250c42f5206ff" gracePeriod=30
Nov 03 19:54:52.610944 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:52.610886    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:52.929338 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:52.929298    5659 generic.go:356] "Generic (PLEG): container finished" podID="87abccc8-39bc-412e-b61e-68a960e5d269" containerID="eacf4383b5dd4280fb9e2fd7696d20c7beeaf0bc13777cb3ef7250c42f5206ff" exitCode=0
Nov 03 19:54:52.929665 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:52.929395    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx" event={"ID":"87abccc8-39bc-412e-b61e-68a960e5d269","Type":"ContainerDied","Data":"eacf4383b5dd4280fb9e2fd7696d20c7beeaf0bc13777cb3ef7250c42f5206ff"}
Nov 03 19:54:53.017617 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.017588    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:53.081876 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.081838    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:53.099381 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.099347    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-maas-api-tls\") pod \"87abccc8-39bc-412e-b61e-68a960e5d269\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") "
Nov 03 19:54:53.099548 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.099444    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-metrics-certs\") pod \"87abccc8-39bc-412e-b61e-68a960e5d269\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") "
Nov 03 19:54:53.099548 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.099497    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87abccc8-39bc-412e-b61e-68a960e5d269-kube-api-access-nlqql\" (UniqueName: \"kubernetes.io/projected/87abccc8-39bc-412e-b61e-68a960e5d269-kube-api-access-nlqql\") pod \"87abccc8-39bc-412e-b61e-68a960e5d269\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") "
Nov 03 19:54:53.099548 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.099521    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87abccc8-39bc-412e-b61e-68a960e5d269-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87abccc8-39bc-412e-b61e-68a960e5d269-openshift-service-ca\") pod \"87abccc8-39bc-412e-b61e-68a960e5d269\" (UID: \"87abccc8-39bc-412e-b61e-68a960e5d269\") "
Nov 03 19:54:53.099922 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.099896    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87abccc8-39bc-412e-b61e-68a960e5d269-openshift-service-ca" pod "87abccc8-39bc-412e-b61e-68a960e5d269" (UID: "87abccc8-39bc-412e-b61e-68a960e5d269"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:54:53.100041 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.100022    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87abccc8-39bc-412e-b61e-68a960e5d269-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:53.101467 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.101444    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-maas-api-tls" pod "87abccc8-39bc-412e-b61e-68a960e5d269" (UID: "87abccc8-39bc-412e-b61e-68a960e5d269"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:54:53.101589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.101509    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87abccc8-39bc-412e-b61e-68a960e5d269-kube-api-access-nlqql" pod "87abccc8-39bc-412e-b61e-68a960e5d269" (UID: "87abccc8-39bc-412e-b61e-68a960e5d269"). InnerVolumeSpecName "kube-api-access-nlqql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:53.101589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.101522    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-metrics-certs" pod "87abccc8-39bc-412e-b61e-68a960e5d269" (UID: "87abccc8-39bc-412e-b61e-68a960e5d269"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:54:53.200847 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.200772    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:53.200847 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.200800    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87abccc8-39bc-412e-b61e-68a960e5d269-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:53.200847 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.200810    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlqql\" (UniqueName: \"kubernetes.io/projected/87abccc8-39bc-412e-b61e-68a960e5d269-kube-api-access-nlqql\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:53.287284 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.287246    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7" podUID="4547bb97-69e8-4103-a263-3284b988223b" containerName="payload-processing" containerID="cri-o://440e3e121f429f9a8302e1c55378e8cb8f7af41bb679ff488e3e570d9bcbd35d" gracePeriod=25
Nov 03 19:54:53.561566 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.561538    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"
Nov 03 19:54:53.606044 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.606012    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4547bb97-69e8-4103-a263-3284b988223b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4547bb97-69e8-4103-a263-3284b988223b-plugins-config-volume\") pod \"4547bb97-69e8-4103-a263-3284b988223b\" (UID: \"4547bb97-69e8-4103-a263-3284b988223b\") "
Nov 03 19:54:53.606352 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.606316    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4547bb97-69e8-4103-a263-3284b988223b-plugins-config-volume" pod "4547bb97-69e8-4103-a263-3284b988223b" (UID: "4547bb97-69e8-4103-a263-3284b988223b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:54:53.606352 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.606327    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4547bb97-69e8-4103-a263-3284b988223b-kube-api-access-xzckh\" (UniqueName: \"kubernetes.io/projected/4547bb97-69e8-4103-a263-3284b988223b-kube-api-access-xzckh\") pod \"4547bb97-69e8-4103-a263-3284b988223b\" (UID: \"4547bb97-69e8-4103-a263-3284b988223b\") "
Nov 03 19:54:53.606981 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.606956    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4547bb97-69e8-4103-a263-3284b988223b-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:53.609813 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.609787    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4547bb97-69e8-4103-a263-3284b988223b-kube-api-access-xzckh" pod "4547bb97-69e8-4103-a263-3284b988223b" (UID: "4547bb97-69e8-4103-a263-3284b988223b"). InnerVolumeSpecName "kube-api-access-xzckh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:53.708159 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.708079    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzckh\" (UniqueName: \"kubernetes.io/projected/4547bb97-69e8-4103-a263-3284b988223b-kube-api-access-xzckh\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:53.736003 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.735967    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:53.937154 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.937122    5659 generic.go:356] "Generic (PLEG): container finished" podID="4547bb97-69e8-4103-a263-3284b988223b" containerID="440e3e121f429f9a8302e1c55378e8cb8f7af41bb679ff488e3e570d9bcbd35d" exitCode=0
Nov 03 19:54:53.937371 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.937169    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"
Nov 03 19:54:53.937371 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.937214    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7" event={"ID":"4547bb97-69e8-4103-a263-3284b988223b","Type":"ContainerDied","Data":"440e3e121f429f9a8302e1c55378e8cb8f7af41bb679ff488e3e570d9bcbd35d"}
Nov 03 19:54:53.937371 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.937269    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7" event={"ID":"4547bb97-69e8-4103-a263-3284b988223b","Type":"ContainerDied","Data":"ee8ff3095f18cc9c997033cad4e653574cf135cf5ba5ef5aa5e450408945d284"}
Nov 03 19:54:53.937371 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.937300    5659 scope.go:122] "RemoveContainer" containerID="440e3e121f429f9a8302e1c55378e8cb8f7af41bb679ff488e3e570d9bcbd35d"
Nov 03 19:54:53.938736 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.938714    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"
Nov 03 19:54:53.938846 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.938730    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx" event={"ID":"87abccc8-39bc-412e-b61e-68a960e5d269","Type":"ContainerDied","Data":"28353adbfc252ab474d2a0f88586902004f3502ff0bd5cbe5d88dc447bac02f2"}
Nov 03 19:54:53.951616 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.951591    5659 scope.go:122] "RemoveContainer" containerID="440e3e121f429f9a8302e1c55378e8cb8f7af41bb679ff488e3e570d9bcbd35d"
Nov 03 19:54:53.951908 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:54:53.951886    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440e3e121f429f9a8302e1c55378e8cb8f7af41bb679ff488e3e570d9bcbd35d\": container with ID starting with 440e3e121f429f9a8302e1c55378e8cb8f7af41bb679ff488e3e570d9bcbd35d not found: ID does not exist" containerID="440e3e121f429f9a8302e1c55378e8cb8f7af41bb679ff488e3e570d9bcbd35d"
Nov 03 19:54:53.951980 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.951915    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440e3e121f429f9a8302e1c55378e8cb8f7af41bb679ff488e3e570d9bcbd35d"} err="failed to get container status \"440e3e121f429f9a8302e1c55378e8cb8f7af41bb679ff488e3e570d9bcbd35d\": rpc error: code = NotFound desc = could not find container \"440e3e121f429f9a8302e1c55378e8cb8f7af41bb679ff488e3e570d9bcbd35d\": container with ID starting with 440e3e121f429f9a8302e1c55378e8cb8f7af41bb679ff488e3e570d9bcbd35d not found: ID does not exist"
Nov 03 19:54:53.951980 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.951933    5659 scope.go:122] "RemoveContainer" containerID="eacf4383b5dd4280fb9e2fd7696d20c7beeaf0bc13777cb3ef7250c42f5206ff"
Nov 03 19:54:53.983126 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.983098    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"]
Nov 03 19:54:53.983126 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.983133    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a2237-85f764d744-xn9t7"]
Nov 03 19:54:53.992977 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:53.992950    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"]
Nov 03 19:54:54.002145 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:54.002117    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0a2237-59fb945c5-d2cwx"]
Nov 03 19:54:54.699638 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:54.699587    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:54.936684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:54.936648    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:55.166064 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:55.166031    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:55.669374 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:55.669345    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:55.924527 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:55.924430    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:55.973907 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:55.973871    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4547bb97-69e8-4103-a263-3284b988223b" path="/var/lib/kubelet/pods/4547bb97-69e8-4103-a263-3284b988223b/volumes"
Nov 03 19:54:55.974660 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:55.974624    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87abccc8-39bc-412e-b61e-68a960e5d269" path="/var/lib/kubelet/pods/87abccc8-39bc-412e-b61e-68a960e5d269/volumes"
Nov 03 19:54:56.397043 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:56.397008    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:56.528195 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:56.528159    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:56.664625 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:56.664528    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:57.208957 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:57.208924    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:57.351145 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:57.351112    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:57.595876 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:57.595752    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:58.395300 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:58.395269    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:58.531192 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:58.531166    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:59.272777 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:59.272742    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:59.964638 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:54:59.964603    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:00.766452 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:00.766414    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:01.566644 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:01.566613    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:01.692452 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:01.692409    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:01.845748 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:01.845656    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:01.974368 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:01.974329    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:02.108799 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:02.108713    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:02.396547 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:02.396448    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:02.622019 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:02.621976    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:03.213453 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:03.213416    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:03.647751 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:03.647660    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:04.421135 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:04.421096    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:04.551719 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:04.551677    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:05.291739 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:05.291700    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:05.426780 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:05.426746    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:05.562457 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:05.562364    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:05.907682 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:05.907603    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:06.213770 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:06.213735    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:06.483047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:06.482963    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:07.346578 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:07.346546    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:08.063141 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:08.063037    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:08.735034 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:08.734994    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:09.160295 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:09.160197    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:09.295162 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:09.295123    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:10.119205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:10.119171    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:10.349901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:10.349844    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:10.943443 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:10.943407    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:11.636883 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:11.636848    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:12.010334 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:12.010302    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:12.139696 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:12.139661    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:12.350297 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:12.350139    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:12.582953 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:12.582915    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:12.978283 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:12.978254    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:13.449291 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:13.449199    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:14.130390 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:14.130352    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:14.802542 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:14.802446    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:15.159812 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:15.159725    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:15.218806 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:55:15.218764    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:55:16.083643 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:16.083583    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:16.653883 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:16.653848    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:16.788332 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:16.788293    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:16.924101 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:16.924021    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:17.236973 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:17.236943    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:17.624727 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:17.624630    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:17.769745 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:17.765159    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:18.660908 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:18.660858    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:19.181124 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.181088    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"]
Nov 03 19:55:19.181998 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.181944    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd" podUID="9d5154a2-122f-4b75-af12-b023995483e2" containerName="maas-api" containerID="cri-o://3b9416d3f5891d26e8529f293459eb7e4bebf0028680020026e34e426aa86cd6" gracePeriod=30
Nov 03 19:55:19.245381 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.245233    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:19.401457 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.401374    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:19.567056 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.566014    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:19.725929 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.725902    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:55:19.796737 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.796650    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:19.850438 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.850397    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-maas-api-tls\") pod \"9d5154a2-122f-4b75-af12-b023995483e2\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") "
Nov 03 19:55:19.850659 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.850539    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5154a2-122f-4b75-af12-b023995483e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5154a2-122f-4b75-af12-b023995483e2-openshift-service-ca\") pod \"9d5154a2-122f-4b75-af12-b023995483e2\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") "
Nov 03 19:55:19.850659 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.850577    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-metrics-certs\") pod \"9d5154a2-122f-4b75-af12-b023995483e2\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") "
Nov 03 19:55:19.850659 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.850610    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5154a2-122f-4b75-af12-b023995483e2-kube-api-access-h8cmw\" (UniqueName: \"kubernetes.io/projected/9d5154a2-122f-4b75-af12-b023995483e2-kube-api-access-h8cmw\") pod \"9d5154a2-122f-4b75-af12-b023995483e2\" (UID: \"9d5154a2-122f-4b75-af12-b023995483e2\") "
Nov 03 19:55:19.850976 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.850947    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5154a2-122f-4b75-af12-b023995483e2-openshift-service-ca" pod "9d5154a2-122f-4b75-af12-b023995483e2" (UID: "9d5154a2-122f-4b75-af12-b023995483e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:19.851265 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.851234    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5154a2-122f-4b75-af12-b023995483e2-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:19.852871 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.852844    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-metrics-certs" pod "9d5154a2-122f-4b75-af12-b023995483e2" (UID: "9d5154a2-122f-4b75-af12-b023995483e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:55:19.853086 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.853060    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-maas-api-tls" pod "9d5154a2-122f-4b75-af12-b023995483e2" (UID: "9d5154a2-122f-4b75-af12-b023995483e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:55:19.853157 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.853095    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5154a2-122f-4b75-af12-b023995483e2-kube-api-access-h8cmw" pod "9d5154a2-122f-4b75-af12-b023995483e2" (UID: "9d5154a2-122f-4b75-af12-b023995483e2"). InnerVolumeSpecName "kube-api-access-h8cmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:19.952321 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.952286    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:19.952321 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.952320    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8cmw\" (UniqueName: \"kubernetes.io/projected/9d5154a2-122f-4b75-af12-b023995483e2-kube-api-access-h8cmw\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:19.952557 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:19.952335    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5154a2-122f-4b75-af12-b023995483e2-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:20.198598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:20.198506    5659 generic.go:356] "Generic (PLEG): container finished" podID="9d5154a2-122f-4b75-af12-b023995483e2" containerID="3b9416d3f5891d26e8529f293459eb7e4bebf0028680020026e34e426aa86cd6" exitCode=0
Nov 03 19:55:20.198598 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:20.198555    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"
Nov 03 19:55:20.199047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:20.198588    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd" event={"ID":"9d5154a2-122f-4b75-af12-b023995483e2","Type":"ContainerDied","Data":"3b9416d3f5891d26e8529f293459eb7e4bebf0028680020026e34e426aa86cd6"}
Nov 03 19:55:20.199047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:20.198648    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd" event={"ID":"9d5154a2-122f-4b75-af12-b023995483e2","Type":"ContainerDied","Data":"13638a0057d50f7d57d1818313f2ac7a476020365b789a393d6502855326477f"}
Nov 03 19:55:20.199047 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:20.198676    5659 scope.go:122] "RemoveContainer" containerID="3b9416d3f5891d26e8529f293459eb7e4bebf0028680020026e34e426aa86cd6"
Nov 03 19:55:20.209002 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:20.208979    5659 scope.go:122] "RemoveContainer" containerID="3b9416d3f5891d26e8529f293459eb7e4bebf0028680020026e34e426aa86cd6"
Nov 03 19:55:20.209295 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:55:20.209275    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9416d3f5891d26e8529f293459eb7e4bebf0028680020026e34e426aa86cd6\": container with ID starting with 3b9416d3f5891d26e8529f293459eb7e4bebf0028680020026e34e426aa86cd6 not found: ID does not exist" containerID="3b9416d3f5891d26e8529f293459eb7e4bebf0028680020026e34e426aa86cd6"
Nov 03 19:55:20.209355 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:20.209304    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9416d3f5891d26e8529f293459eb7e4bebf0028680020026e34e426aa86cd6"} err="failed to get container status \"3b9416d3f5891d26e8529f293459eb7e4bebf0028680020026e34e426aa86cd6\": rpc error: code = NotFound desc = could not find container \"3b9416d3f5891d26e8529f293459eb7e4bebf0028680020026e34e426aa86cd6\": container with ID starting with 3b9416d3f5891d26e8529f293459eb7e4bebf0028680020026e34e426aa86cd6 not found: ID does not exist"
Nov 03 19:55:20.220345 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:20.220301    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"]
Nov 03 19:55:20.224318 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:20.224292    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-tg7vd"]
Nov 03 19:55:20.628773 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:20.628744    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:21.444742 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:21.444698    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:21.970555 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:21.970512    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5154a2-122f-4b75-af12-b023995483e2" path="/var/lib/kubelet/pods/9d5154a2-122f-4b75-af12-b023995483e2/volumes"
Nov 03 19:55:22.134163 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:22.134114    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:22.838034 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:22.837999    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:23.669562 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:23.669528    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:23.958078 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:23.957996    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:24.767099 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:24.767061    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:25.561229 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:25.561200    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:26.053628 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:26.053598    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:26.338388 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:26.338297    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:26.496497 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:26.496452    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:26.817218 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:26.817181    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:26.952392 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:26.952356    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:27.300822 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:27.300781    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:27.440143 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:27.440113    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:28.385120 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:28.385082    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:29.041111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:29.041069    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:29.828956 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:29.828883    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:30.194319 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:30.194188    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:30.316740 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:30.316687    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:30.777654 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:30.777605    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:31.393259 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:31.393224    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:31.536776 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:31.536740    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:31.673690 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:31.673595    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:31.809624 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:31.809592    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:32.558375 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:32.558335    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:33.518402 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:33.518373    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:34.153855 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:34.153823    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:34.911768 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:34.911740    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:35.049524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:35.049474    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:35.182190 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:35.182077    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:35.510159 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:35.510127    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:36.016853 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:36.016820    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:36.893358 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:36.893329    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:37.631185 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:37.631153    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:38.346217 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:38.346163    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:38.950884 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:38.950788    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:39.682281 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:39.682240    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:40.263868 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:40.263835    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:40.397698 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:40.397664    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:40.533662 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:40.533578    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:40.849771 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:40.849587    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:40.998768 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:40.998737    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:41.134118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:41.134024    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:41.459394 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:41.459360    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:41.586987 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:41.586953    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:42.163864 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:42.163833    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:42.952506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:42.952441    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:43.940838 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:43.940798    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:44.619550 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:44.619513    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:44.773341 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:44.773310    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:44.916100 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:44.916019    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:45.048443 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:45.048408    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:45.220211 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:55:45.220163    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:55:46.113941 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:46.113904    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:46.272429 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:46.272395    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:46.891369 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:46.891325    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:47.845384 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:47.845349    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:47.991044 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:47.991011    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:48.134440 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:48.134090    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:49.250200 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:49.250164    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:50.138364 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:50.138330    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:51.162622 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:51.162567    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:51.955254 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:51.955218    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:52.651094 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.651046    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"]
Nov 03 19:55:52.652053 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.652029    5659 cpu_manager.go:414] "containerMap: removing container" podUID="9d5154a2-122f-4b75-af12-b023995483e2" containerName="maas-api"
Nov 03 19:55:52.652231 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.652206    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5154a2-122f-4b75-af12-b023995483e2" containerName="maas-api"
Nov 03 19:55:52.652353 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.652270    5659 cpu_manager.go:414] "containerMap: removing container" podUID="4547bb97-69e8-4103-a263-3284b988223b" containerName="payload-processing"
Nov 03 19:55:52.652353 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.652281    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4547bb97-69e8-4103-a263-3284b988223b" containerName="payload-processing"
Nov 03 19:55:52.652353 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.652293    5659 cpu_manager.go:414] "containerMap: removing container" podUID="87abccc8-39bc-412e-b61e-68a960e5d269" containerName="maas-api"
Nov 03 19:55:52.652353 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.652301    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87abccc8-39bc-412e-b61e-68a960e5d269" containerName="maas-api"
Nov 03 19:55:52.652652 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.652460    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="4547bb97-69e8-4103-a263-3284b988223b" containerName="payload-processing"
Nov 03 19:55:52.652652 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.652473    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="87abccc8-39bc-412e-b61e-68a960e5d269" containerName="maas-api"
Nov 03 19:55:52.652652 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.652509    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5154a2-122f-4b75-af12-b023995483e2" containerName="maas-api"
Nov 03 19:55:52.656551 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.656523    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:52.662283 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.662264    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-389e6d-serving-cert\""
Nov 03 19:55:52.667771 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.667741    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"]
Nov 03 19:55:52.716164 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.716130    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-metrics-certs\") pod \"maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:52.716335 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.716169    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svd2g\" (UniqueName: \"kubernetes.io/projected/82dd5715-f5dc-492e-8764-0c8fd945fbe7-kube-api-access-svd2g\") pod \"maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:52.716335 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.716274    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82dd5715-f5dc-492e-8764-0c8fd945fbe7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:52.716416 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.716376    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:52.731109 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.731071    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:52.818661 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.818585    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:52.819698 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.819674    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-metrics-certs\") pod \"maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:52.820310 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.820290    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svd2g\" (UniqueName: \"kubernetes.io/projected/82dd5715-f5dc-492e-8764-0c8fd945fbe7-kube-api-access-svd2g\") pod \"maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:52.820789 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.820771    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82dd5715-f5dc-492e-8764-0c8fd945fbe7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:52.822771 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.821476    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82dd5715-f5dc-492e-8764-0c8fd945fbe7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:52.823552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.823327    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:52.824084 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.824063    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-metrics-certs\") pod \"maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:52.834144 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.834108    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svd2g\" (UniqueName: \"kubernetes.io/projected/82dd5715-f5dc-492e-8764-0c8fd945fbe7-kube-api-access-svd2g\") pod \"maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:52.855879 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.855845    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:52.976212 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:52.976176    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:53.310173 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:53.310073    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"]
Nov 03 19:55:53.490701 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:53.490665    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f" event={"ID":"82dd5715-f5dc-492e-8764-0c8fd945fbe7","Type":"ContainerStarted","Data":"5dfe04f856eacaad3a60df8a77d0cb9d8b32b8d12e3ac8dca950d15fe3703bd2"}
Nov 03 19:55:53.650254 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:53.650151    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:53.795505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:53.792062    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:54.500533 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:54.500476    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f" event={"ID":"82dd5715-f5dc-492e-8764-0c8fd945fbe7","Type":"ContainerStarted","Data":"723beb9bd24d6e361d98a72beef0950cc704b5d4bce58b77c6d39c4bbb248a76"}
Nov 03 19:55:54.500717 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:54.500575    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:55:54.521494 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:54.521439    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f" podStartSLOduration=2.112612826 podStartE2EDuration="2.521424698s" totalImagesPullingTime="408.811872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:54.519511001 +0000 UTC m=+1106.674413421" watchObservedRunningTime="2026-11-03 19:55:54.521424698 +0000 UTC m=+1106.676327121"
Nov 03 19:55:54.723260 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:54.723230    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:55.380905 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:55.380871    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:55.574191 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:55.574150    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:56.029265 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:56.029231    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:56.457835 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:56.457800    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:56.601769 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:56.601312    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:56.740931 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:56.740850    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:57.333681 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:57.333646    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:57.916836 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:57.916797    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:58.148026 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:58.147991    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:59.096791 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:55:59.096748    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:00.118060 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:00.118028    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:00.512609 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:00.512582    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 19:56:00.926215 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:00.926127    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:01.112870 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:01.112841    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:01.247787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:01.247743    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:01.396656 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:01.396621    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:02.415866 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.415835    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:02.473611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.473576    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"]
Nov 03 19:56:02.477655 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.477633    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"
Nov 03 19:56:02.484909 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.484881    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b0f9d6\""
Nov 03 19:56:02.485072 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.484940    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b0f9d6-dockercfg-gcptq\""
Nov 03 19:56:02.489519 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.489457    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"]
Nov 03 19:56:02.510583 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.510546    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"]
Nov 03 19:56:02.516179 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.516151    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"
Nov 03 19:56:02.518696 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.518669    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"]
Nov 03 19:56:02.550815 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.550782    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:02.635505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.632551    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kf2z\" (UniqueName: \"kubernetes.io/projected/b951e973-55bb-4eb5-a882-8a0eef28572f-kube-api-access-8kf2z\") pod \"payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph\" (UID: \"b951e973-55bb-4eb5-a882-8a0eef28572f\") " pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"
Nov 03 19:56:02.635505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.632623    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7f216d-bf8d-4e20-811e-19cc85424099-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds\" (UID: \"1d7f216d-bf8d-4e20-811e-19cc85424099\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"
Nov 03 19:56:02.635505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.632721    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z9dv\" (UniqueName: \"kubernetes.io/projected/1d7f216d-bf8d-4e20-811e-19cc85424099-kube-api-access-8z9dv\") pod \"payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds\" (UID: \"1d7f216d-bf8d-4e20-811e-19cc85424099\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"
Nov 03 19:56:02.635505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.632818    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b951e973-55bb-4eb5-a882-8a0eef28572f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph\" (UID: \"b951e973-55bb-4eb5-a882-8a0eef28572f\") " pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"
Nov 03 19:56:02.734820 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.734780    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b951e973-55bb-4eb5-a882-8a0eef28572f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph\" (UID: \"b951e973-55bb-4eb5-a882-8a0eef28572f\") " pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"
Nov 03 19:56:02.735250 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.735230    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kf2z\" (UniqueName: \"kubernetes.io/projected/b951e973-55bb-4eb5-a882-8a0eef28572f-kube-api-access-8kf2z\") pod \"payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph\" (UID: \"b951e973-55bb-4eb5-a882-8a0eef28572f\") " pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"
Nov 03 19:56:02.735402 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.735386    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7f216d-bf8d-4e20-811e-19cc85424099-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds\" (UID: \"1d7f216d-bf8d-4e20-811e-19cc85424099\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"
Nov 03 19:56:02.735628 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.735609    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9dv\" (UniqueName: \"kubernetes.io/projected/1d7f216d-bf8d-4e20-811e-19cc85424099-kube-api-access-8z9dv\") pod \"payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds\" (UID: \"1d7f216d-bf8d-4e20-811e-19cc85424099\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"
Nov 03 19:56:02.737271 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.737244    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b951e973-55bb-4eb5-a882-8a0eef28572f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph\" (UID: \"b951e973-55bb-4eb5-a882-8a0eef28572f\") " pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"
Nov 03 19:56:02.738510 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.738471    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7f216d-bf8d-4e20-811e-19cc85424099-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds\" (UID: \"1d7f216d-bf8d-4e20-811e-19cc85424099\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"
Nov 03 19:56:02.758565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.754028    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kf2z\" (UniqueName: \"kubernetes.io/projected/b951e973-55bb-4eb5-a882-8a0eef28572f-kube-api-access-8kf2z\") pod \"payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph\" (UID: \"b951e973-55bb-4eb5-a882-8a0eef28572f\") " pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"
Nov 03 19:56:02.776305 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.776268    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z9dv\" (UniqueName: \"kubernetes.io/projected/1d7f216d-bf8d-4e20-811e-19cc85424099-kube-api-access-8z9dv\") pod \"payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds\" (UID: \"1d7f216d-bf8d-4e20-811e-19cc85424099\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"
Nov 03 19:56:02.795768 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.795732    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"
Nov 03 19:56:02.832357 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.832318    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"
Nov 03 19:56:02.857509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.854720    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"]
Nov 03 19:56:02.952556 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:02.952072    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:03.133533 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:03.133503    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"]
Nov 03 19:56:03.134297 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:56:03.134269    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7f216d_bf8d_4e20_811e_19cc85424099.slice/crio-f6320929b63176396601b7087ede7a54d1472b1e0237f3473bf73f522012b7dd WatchSource:0}: Error finding container f6320929b63176396601b7087ede7a54d1472b1e0237f3473bf73f522012b7dd: Status 404 returned error can't find the container with id f6320929b63176396601b7087ede7a54d1472b1e0237f3473bf73f522012b7dd
Nov 03 19:56:03.161407 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:03.161383    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"]
Nov 03 19:56:03.170867 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:56:03.170840    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb951e973_55bb_4eb5_a882_8a0eef28572f.slice/crio-46a8ef63a0384a2abfd2cfcd3a919d0ef2cf11221f4904fd8d9943c6c0c31ade WatchSource:0}: Error finding container 46a8ef63a0384a2abfd2cfcd3a919d0ef2cf11221f4904fd8d9943c6c0c31ade: Status 404 returned error can't find the container with id 46a8ef63a0384a2abfd2cfcd3a919d0ef2cf11221f4904fd8d9943c6c0c31ade
Nov 03 19:56:03.573427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:03.573388    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds" event={"ID":"1d7f216d-bf8d-4e20-811e-19cc85424099","Type":"ContainerStarted","Data":"036d29b83129106faf86cc2a0f573eb97e45166f477fdef8cce4d5fa21c2dc0f"}
Nov 03 19:56:03.573918 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:03.573456    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds" event={"ID":"1d7f216d-bf8d-4e20-811e-19cc85424099","Type":"ContainerStarted","Data":"f6320929b63176396601b7087ede7a54d1472b1e0237f3473bf73f522012b7dd"}
Nov 03 19:56:03.573918 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:03.573584    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"
Nov 03 19:56:03.575184 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:03.575155    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph" event={"ID":"b951e973-55bb-4eb5-a882-8a0eef28572f","Type":"ContainerStarted","Data":"83e61ef9001e7e9013d40d5a7879087b70e4d3b48bd88b878d4853824ceeac9d"}
Nov 03 19:56:03.575338 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:03.575205    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph" event={"ID":"b951e973-55bb-4eb5-a882-8a0eef28572f","Type":"ContainerStarted","Data":"46a8ef63a0384a2abfd2cfcd3a919d0ef2cf11221f4904fd8d9943c6c0c31ade"}
Nov 03 19:56:03.575338 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:03.575294    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"
Nov 03 19:56:03.593961 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:03.593915    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds" podStartSLOduration=1.593902029 podStartE2EDuration="1.593902029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:03.592102646 +0000 UTC m=+1115.747005065" watchObservedRunningTime="2026-11-03 19:56:03.593902029 +0000 UTC m=+1115.748804488"
Nov 03 19:56:03.614763 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:03.614710    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph" podStartSLOduration=1.614697984 podStartE2EDuration="1.614697984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:03.611608385 +0000 UTC m=+1115.766510831" watchObservedRunningTime="2026-11-03 19:56:03.614697984 +0000 UTC m=+1115.769600403"
Nov 03 19:56:03.754073 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:03.754019    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:04.808537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:04.808470    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:04.942919 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:04.942884    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:05.090561 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:05.086694    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:05.746862 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:05.740422    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:06.433340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:06.433291    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:07.070803 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:07.070760    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:07.850066 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:07.850034    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:08.351344 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:08.351300    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:08.668458 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:08.668366    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:08.825623 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:08.825571    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:09.285360 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:09.285317    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:09.615213 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:09.615130    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:09.845034 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:09.844998    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:10.065506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:10.048896    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:10.192872 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:10.192841    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:10.349388 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:10.349283    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:10.808506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:10.807299    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:11.747344 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:11.747303    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:12.660843 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:12.660813    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:13.043213 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:13.043175    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:13.182813 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:13.182779    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:13.328182 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:13.328092    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:13.506529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:13.492964    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:13.967627 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:13.967553    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:14.299003 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.298921    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:14.349218 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.349185    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"]
Nov 03 19:56:14.354630 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.354604    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"
Nov 03 19:56:14.359200 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.359170    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"]
Nov 03 19:56:14.363075 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.363049    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b34f6c-dockercfg-7cwhh\""
Nov 03 19:56:14.419017 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.418989    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b34f6c\""
Nov 03 19:56:14.434881 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.434845    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:14.508120 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.507595    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339756bb-9cd5-4a08-a142-93ea4affded0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq\" (UID: \"339756bb-9cd5-4a08-a142-93ea4affded0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"
Nov 03 19:56:14.508120 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.507694    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576qj\" (UniqueName: \"kubernetes.io/projected/339756bb-9cd5-4a08-a142-93ea4affded0-kube-api-access-576qj\") pod \"payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq\" (UID: \"339756bb-9cd5-4a08-a142-93ea4affded0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"
Nov 03 19:56:14.615604 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.615507    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339756bb-9cd5-4a08-a142-93ea4affded0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq\" (UID: \"339756bb-9cd5-4a08-a142-93ea4affded0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"
Nov 03 19:56:14.620507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.620466    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576qj\" (UniqueName: \"kubernetes.io/projected/339756bb-9cd5-4a08-a142-93ea4affded0-kube-api-access-576qj\") pod \"payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq\" (UID: \"339756bb-9cd5-4a08-a142-93ea4affded0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"
Nov 03 19:56:14.621235 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.616637    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339756bb-9cd5-4a08-a142-93ea4affded0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq\" (UID: \"339756bb-9cd5-4a08-a142-93ea4affded0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"
Nov 03 19:56:14.630904 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.626676    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"
Nov 03 19:56:14.634501 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.631160    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"
Nov 03 19:56:14.655438 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.655409    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-576qj\" (UniqueName: \"kubernetes.io/projected/339756bb-9cd5-4a08-a142-93ea4affded0-kube-api-access-576qj\") pod \"payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq\" (UID: \"339756bb-9cd5-4a08-a142-93ea4affded0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"
Nov 03 19:56:14.690305 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.690270    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"
Nov 03 19:56:14.706518 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.706451    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:14.836960 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:14.836901    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:15.085506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:15.084582    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"]
Nov 03 19:56:15.120154 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:56:15.120124    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339756bb_9cd5_4a08_a142_93ea4affded0.slice/crio-0794e7537b5666e4a41e1f56ec0fbc46b51746c317e43af7eea3a30463450be7 WatchSource:0}: Error finding container 0794e7537b5666e4a41e1f56ec0fbc46b51746c317e43af7eea3a30463450be7: Status 404 returned error can't find the container with id 0794e7537b5666e4a41e1f56ec0fbc46b51746c317e43af7eea3a30463450be7
Nov 03 19:56:15.228741 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:56:15.228712    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:56:15.547497 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:15.547452    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:15.700135 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:15.700097    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq" event={"ID":"339756bb-9cd5-4a08-a142-93ea4affded0","Type":"ContainerStarted","Data":"8ffe099e5ee0671eb48b8f918890f5fdcba9476aaebe58a1aaa9e6141bb8912e"}
Nov 03 19:56:15.700326 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:15.700158    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq" event={"ID":"339756bb-9cd5-4a08-a142-93ea4affded0","Type":"ContainerStarted","Data":"0794e7537b5666e4a41e1f56ec0fbc46b51746c317e43af7eea3a30463450be7"}
Nov 03 19:56:15.700326 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:15.700312    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"
Nov 03 19:56:15.725033 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:15.724977    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq" podStartSLOduration=1.724965421 podStartE2EDuration="1.724965421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:15.722231862 +0000 UTC m=+1127.877134282" watchObservedRunningTime="2026-11-03 19:56:15.724965421 +0000 UTC m=+1127.879867840"
Nov 03 19:56:16.631270 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:16.631241    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:17.321097 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:17.321024    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:17.894733 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:17.894670    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8" podUID="243a0d94-99af-4a65-b019-64f05ebe95a5" containerName="main" containerID="cri-o://a2fd9ee51d2c29ec9431e12ece3d983f507434f0c9d449b55b86f4770aa55ec3" gracePeriod=45
Nov 03 19:56:18.034572 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.034521    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8" podUID="243a0d94-99af-4a65-b019-64f05ebe95a5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": dial tcp 10.133.0.74:8000: connect: connection refused"
Nov 03 19:56:18.063510 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.063464    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:18.223877 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.223848    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:56:18.284602 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.284570    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243a0d94-99af-4a65-b019-64f05ebe95a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/243a0d94-99af-4a65-b019-64f05ebe95a5-tls-certs\") pod \"243a0d94-99af-4a65-b019-64f05ebe95a5\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") "
Nov 03 19:56:18.284767 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.284650    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-home\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-home\") pod \"243a0d94-99af-4a65-b019-64f05ebe95a5\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") "
Nov 03 19:56:18.284767 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.284704    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-model-cache\") pod \"243a0d94-99af-4a65-b019-64f05ebe95a5\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") "
Nov 03 19:56:18.284767 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.284732    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-dshm\") pod \"243a0d94-99af-4a65-b019-64f05ebe95a5\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") "
Nov 03 19:56:18.284918 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.284826    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243a0d94-99af-4a65-b019-64f05ebe95a5-kube-api-access-dxzmw\" (UniqueName: \"kubernetes.io/projected/243a0d94-99af-4a65-b019-64f05ebe95a5-kube-api-access-dxzmw\") pod \"243a0d94-99af-4a65-b019-64f05ebe95a5\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") "
Nov 03 19:56:18.285841 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.285808    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-home" pod "243a0d94-99af-4a65-b019-64f05ebe95a5" (UID: "243a0d94-99af-4a65-b019-64f05ebe95a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:56:18.286022 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.286001    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-model-cache" pod "243a0d94-99af-4a65-b019-64f05ebe95a5" (UID: "243a0d94-99af-4a65-b019-64f05ebe95a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:56:18.289866 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.289835    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243a0d94-99af-4a65-b019-64f05ebe95a5-kube-api-access-dxzmw" pod "243a0d94-99af-4a65-b019-64f05ebe95a5" (UID: "243a0d94-99af-4a65-b019-64f05ebe95a5"). InnerVolumeSpecName "kube-api-access-dxzmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:18.289984 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.289931    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-dshm" pod "243a0d94-99af-4a65-b019-64f05ebe95a5" (UID: "243a0d94-99af-4a65-b019-64f05ebe95a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:56:18.291400 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.291374    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243a0d94-99af-4a65-b019-64f05ebe95a5-tls-certs" pod "243a0d94-99af-4a65-b019-64f05ebe95a5" (UID: "243a0d94-99af-4a65-b019-64f05ebe95a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:56:18.385940 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.385893    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-tmp-dir\") pod \"243a0d94-99af-4a65-b019-64f05ebe95a5\" (UID: \"243a0d94-99af-4a65-b019-64f05ebe95a5\") "
Nov 03 19:56:18.386186 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.386161    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-tmp-dir" pod "243a0d94-99af-4a65-b019-64f05ebe95a5" (UID: "243a0d94-99af-4a65-b019-64f05ebe95a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:56:18.386385 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.386366    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:18.386456 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.386391    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:18.386456 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.386406    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxzmw\" (UniqueName: \"kubernetes.io/projected/243a0d94-99af-4a65-b019-64f05ebe95a5-kube-api-access-dxzmw\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:18.386456 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.386448    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:18.386639 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.386462    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243a0d94-99af-4a65-b019-64f05ebe95a5-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:18.386639 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.386476    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243a0d94-99af-4a65-b019-64f05ebe95a5-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:18.734175 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.734139    5659 generic.go:356] "Generic (PLEG): container finished" podID="243a0d94-99af-4a65-b019-64f05ebe95a5" containerID="a2fd9ee51d2c29ec9431e12ece3d983f507434f0c9d449b55b86f4770aa55ec3" exitCode=0
Nov 03 19:56:18.734372 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.734184    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"
Nov 03 19:56:18.734372 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.734201    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8" event={"ID":"243a0d94-99af-4a65-b019-64f05ebe95a5","Type":"ContainerDied","Data":"a2fd9ee51d2c29ec9431e12ece3d983f507434f0c9d449b55b86f4770aa55ec3"}
Nov 03 19:56:18.734372 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.734240    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8" event={"ID":"243a0d94-99af-4a65-b019-64f05ebe95a5","Type":"ContainerDied","Data":"8a4acc9948316f56b855489b56365f3787a65e44f5149e10bc7f9789d6a219d8"}
Nov 03 19:56:18.734372 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.734266    5659 scope.go:122] "RemoveContainer" containerID="a2fd9ee51d2c29ec9431e12ece3d983f507434f0c9d449b55b86f4770aa55ec3"
Nov 03 19:56:18.746150 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.746131    5659 scope.go:122] "RemoveContainer" containerID="a2fd9ee51d2c29ec9431e12ece3d983f507434f0c9d449b55b86f4770aa55ec3"
Nov 03 19:56:18.746401 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:56:18.746383    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2fd9ee51d2c29ec9431e12ece3d983f507434f0c9d449b55b86f4770aa55ec3\": container with ID starting with a2fd9ee51d2c29ec9431e12ece3d983f507434f0c9d449b55b86f4770aa55ec3 not found: ID does not exist" containerID="a2fd9ee51d2c29ec9431e12ece3d983f507434f0c9d449b55b86f4770aa55ec3"
Nov 03 19:56:18.746458 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.746411    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2fd9ee51d2c29ec9431e12ece3d983f507434f0c9d449b55b86f4770aa55ec3"} err="failed to get container status \"a2fd9ee51d2c29ec9431e12ece3d983f507434f0c9d449b55b86f4770aa55ec3\": rpc error: code = NotFound desc = could not find container \"a2fd9ee51d2c29ec9431e12ece3d983f507434f0c9d449b55b86f4770aa55ec3\": container with ID starting with a2fd9ee51d2c29ec9431e12ece3d983f507434f0c9d449b55b86f4770aa55ec3 not found: ID does not exist"
Nov 03 19:56:18.757731 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.756850    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:18.771766 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.771732    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"]
Nov 03 19:56:18.776511 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:18.776464    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d8d238bf/e2e-lifecycle-model-d8d238bf-kserve-78fff9cdf7-d9rf8"]
Nov 03 19:56:19.390635 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:19.390589    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:19.969133 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:19.969097    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243a0d94-99af-4a65-b019-64f05ebe95a5" path="/var/lib/kubelet/pods/243a0d94-99af-4a65-b019-64f05ebe95a5/volumes"
Nov 03 19:56:20.047131 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:20.047099    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:21.131610 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:21.131579    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:21.878188 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:21.878152    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:22.625101 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:22.625069    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:24.316655 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:24.316624    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:24.454945 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:24.454912    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:25.243880 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:25.243852    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:25.900805 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:25.900769    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:26.622996 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:26.622964    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:26.712381 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:26.712352    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"
Nov 03 19:56:27.560714 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:27.560659    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:28.161531 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:28.161501    5659 scope.go:122] "RemoveContainer" containerID="caff2f12e1d5ae91e0e0368002501fde20510066f8a575c0302f4893e2bf9a29"
Nov 03 19:56:28.546085 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:28.546053    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:29.322825 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:29.322787    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:29.958695 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:29.958657    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:30.729706 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:30.729654    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:31.444243 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:31.444196    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:32.229032 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:32.228997    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:33.183358 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:33.183318    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:34.139716 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:34.139687    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:34.973074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:34.973028    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:35.566195 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:35.566157    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:35.730870 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:35.730838    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:36.113623 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:36.113592    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:36.444408 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:36.444288    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:36.843086 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:36.843048    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:37.053729 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.053700    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:37.221700 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.221670    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"]
Nov 03 19:56:37.222500 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.222472    5659 cpu_manager.go:414] "containerMap: removing container" podUID="243a0d94-99af-4a65-b019-64f05ebe95a5" containerName="main"
Nov 03 19:56:37.222557 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.222502    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243a0d94-99af-4a65-b019-64f05ebe95a5" containerName="main"
Nov 03 19:56:37.222667 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.222656    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="243a0d94-99af-4a65-b019-64f05ebe95a5" containerName="main"
Nov 03 19:56:37.228299 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.228270    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.232355 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.232328    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"]
Nov 03 19:56:37.239508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.239472    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3d1db1-openshift-default-dockercfg-2ndm2\""
Nov 03 19:56:37.282254 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.282202    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:37.299133 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.299094    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-certs\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.299133 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.299127    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-data\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.299387 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.299313    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.299387 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.299347    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49ffc14-16a0-495d-954f-a524e0d4a81a-istiod-ca-cert\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.299526 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.299407    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5shrt\" (UniqueName: \"kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-kube-api-access-5shrt\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.299526 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.299437    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-envoy\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.299526 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.299472    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-token\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.299700 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.299550    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-credential-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.299700 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.299571    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-podinfo\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.422505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.422129    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-certs\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.422505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.422178    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-data\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.422505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.422312    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.422505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.422339    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49ffc14-16a0-495d-954f-a524e0d4a81a-istiod-ca-cert\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.422505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.422387    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5shrt\" (UniqueName: \"kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-kube-api-access-5shrt\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.422505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.422418    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-envoy\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.422505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.422441    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-token\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.423013 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.422543    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-credential-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.423013 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.422577    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-podinfo\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.445559 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.445341    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49ffc14-16a0-495d-954f-a524e0d4a81a-istiod-ca-cert\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.445831 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.445800    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-certs\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.446079 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.446058    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-data\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.446323 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.446300    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.447427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.447401    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-credential-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.473977 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.473900    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5shrt\" (UniqueName: \"kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-kube-api-access-5shrt\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.474661 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.474582    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-token\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.479134 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.479109    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-podinfo\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.480799 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.480778    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-envoy\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.503896 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.503867    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:37.548871 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.548839    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:37.932411 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.931186    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"]
Nov 03 19:56:37.949146 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.948528    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:56:37.949146 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.948602    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:56:37.949146 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.948633    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:56:37.973897 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:37.969406    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:38.139279 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:38.138691    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:38.300767 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:38.300736    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:38.552776 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:38.552691    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:38.909363 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:38.909264    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" event={"ID":"a49ffc14-16a0-495d-954f-a524e0d4a81a","Type":"ContainerStarted","Data":"4fa08e00988778f86683bc137aa8a81af9a8e5dad6f8cbcac1803a13915ba093"}
Nov 03 19:56:38.909363 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:38.909333    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" event={"ID":"a49ffc14-16a0-495d-954f-a524e0d4a81a","Type":"ContainerStarted","Data":"d1f1301b4c2af1319db1626d2ea8a2c77b8ba42793be339cd356982b122ff999"}
Nov 03 19:56:38.935344 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:38.935276    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" podStartSLOduration=1.935256216 podStartE2EDuration="1.935256216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:38.93206273 +0000 UTC m=+1151.086965151" watchObservedRunningTime="2026-11-03 19:56:38.935256216 +0000 UTC m=+1151.090158636"
Nov 03 19:56:38.936953 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:38.936922    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:39.294516 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.294228    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"]
Nov 03 19:56:39.304188 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.304155    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"]
Nov 03 19:56:39.304375 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.304253    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.317530 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.307807    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9a6d4b-openshift-default-dockercfg-7bbp5\""
Nov 03 19:56:39.335216 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.335163    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:39.403654 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.403564    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/561ecf04-5f23-49de-8881-639fc816fa81-istiod-ca-cert\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.403654 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.403651    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-socket\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.403899 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.403735    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-certs\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.403899 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.403764    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-istio-token\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.404002 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.403916    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-envoy\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.404053 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.404028    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-data\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.404106 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.404071    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pvst\" (UniqueName: \"kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-kube-api-access-8pvst\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.404162 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.404117    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/561ecf04-5f23-49de-8881-639fc816fa81-istio-podinfo\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.404210 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.404166    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-credential-socket\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.475054 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.475018    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:39.506117 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.505782    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-certs\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.506117 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.505848    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-istio-token\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.506117 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.505982    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-envoy\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.506117 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.506073    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-data\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.506117 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.506112    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pvst\" (UniqueName: \"kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-kube-api-access-8pvst\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.506553 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.506145    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/561ecf04-5f23-49de-8881-639fc816fa81-istio-podinfo\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.506553 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.506182    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-credential-socket\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.506553 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.506230    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/561ecf04-5f23-49de-8881-639fc816fa81-istiod-ca-cert\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.506553 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.506271    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-socket\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.511258 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.506779    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-socket\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.511570 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.511346    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/561ecf04-5f23-49de-8881-639fc816fa81-istiod-ca-cert\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.520503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.512230    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-certs\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.520503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.512595    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-data\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.520503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.512821    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-credential-socket\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.554554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.552826    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:39.555637 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.555567    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pvst\" (UniqueName: \"kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-kube-api-access-8pvst\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.557992 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.557940    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-istio-token\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.558355 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.558315    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-envoy\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.558355 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.558324    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/561ecf04-5f23-49de-8881-639fc816fa81-istio-podinfo\") pod \"e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") " pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.599641 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.599606    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:39.619430 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.614774    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:39.624037 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.624004    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:39.918638 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.918548    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:39.919828 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:39.919803    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:56:40.043367 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:40.043342    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"]
Nov 03 19:56:40.044147 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:40.044112    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:56:40.044245 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:40.044177    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:56:40.044245 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:40.044206    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:56:40.630517 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:40.625532    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:40.926500 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:40.926389    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" event={"ID":"561ecf04-5f23-49de-8881-639fc816fa81","Type":"ContainerStarted","Data":"a6d4f53c0f90986fe14e3c56f54098237bbb171256da82b36b7f36ca7323de5e"}
Nov 03 19:56:40.926500 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:40.926449    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" event={"ID":"561ecf04-5f23-49de-8881-639fc816fa81","Type":"ContainerStarted","Data":"17a176f9c75dfb0d881f9394d08d23477ebc8d99955395ab7e0765519751d84d"}
Nov 03 19:56:40.981548 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:40.980814    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" podStartSLOduration=1.980798965 podStartE2EDuration="1.980798965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:40.973122431 +0000 UTC m=+1153.128024851" watchObservedRunningTime="2026-11-03 19:56:40.980798965 +0000 UTC m=+1153.135701383"
Nov 03 19:56:41.024372 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:41.024341    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:41.159036 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:41.158990    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:41.329517 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:41.328331    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:41.481276 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:41.481237    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:41.625012 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:41.624929    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:41.630817 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:41.630792    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:41.934164 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:41.934074    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:41.935025 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:41.935001    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:56:42.252631 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:42.252580    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:42.533293 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:42.533206    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:43.018802 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:43.018770    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:43.261505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:43.250759    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:43.839287 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:43.839255    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:44.192660 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.192580    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:44.221425 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.221394    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"]
Nov 03 19:56:44.225945 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.225925    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"
Nov 03 19:56:44.238826 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.238803    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9b3619\""
Nov 03 19:56:44.249788 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.249758    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"]
Nov 03 19:56:44.273423 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.273391    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glj4c\" (UniqueName: \"kubernetes.io/projected/529d2c30-6f5e-4f21-a959-66b2f5904229-kube-api-access-glj4c\") pod \"payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6\" (UID: \"529d2c30-6f5e-4f21-a959-66b2f5904229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"
Nov 03 19:56:44.274946 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.274223    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529d2c30-6f5e-4f21-a959-66b2f5904229-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6\" (UID: \"529d2c30-6f5e-4f21-a959-66b2f5904229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"
Nov 03 19:56:44.338940 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.338905    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:44.382910 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.382859    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529d2c30-6f5e-4f21-a959-66b2f5904229-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6\" (UID: \"529d2c30-6f5e-4f21-a959-66b2f5904229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"
Nov 03 19:56:44.398408 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.398371    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glj4c\" (UniqueName: \"kubernetes.io/projected/529d2c30-6f5e-4f21-a959-66b2f5904229-kube-api-access-glj4c\") pod \"payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6\" (UID: \"529d2c30-6f5e-4f21-a959-66b2f5904229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"
Nov 03 19:56:44.413690 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.383952    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529d2c30-6f5e-4f21-a959-66b2f5904229-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6\" (UID: \"529d2c30-6f5e-4f21-a959-66b2f5904229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"
Nov 03 19:56:44.466263 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.466230    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glj4c\" (UniqueName: \"kubernetes.io/projected/529d2c30-6f5e-4f21-a959-66b2f5904229-kube-api-access-glj4c\") pod \"payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6\" (UID: \"529d2c30-6f5e-4f21-a959-66b2f5904229\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"
Nov 03 19:56:44.510007 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.509972    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:44.548843 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.548806    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"
Nov 03 19:56:44.652951 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.652853    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:44.857186 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:44.857157    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"]
Nov 03 19:56:44.857911 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:56:44.857886    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529d2c30_6f5e_4f21_a959_66b2f5904229.slice/crio-6f574d1c23a616391129fdf338a62cc5aaae94cb0660facb91fc253a70c37c71 WatchSource:0}: Error finding container 6f574d1c23a616391129fdf338a62cc5aaae94cb0660facb91fc253a70c37c71: Status 404 returned error can't find the container with id 6f574d1c23a616391129fdf338a62cc5aaae94cb0660facb91fc253a70c37c71
Nov 03 19:56:45.054264 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.054229    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6" event={"ID":"529d2c30-6f5e-4f21-a959-66b2f5904229","Type":"ContainerStarted","Data":"5d3b9a422366b6eb79c6af1d86b132f9a60d029f2612f3058843f5b53f3180cc"}
Nov 03 19:56:45.054442 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.054284    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6" event={"ID":"529d2c30-6f5e-4f21-a959-66b2f5904229","Type":"ContainerStarted","Data":"6f574d1c23a616391129fdf338a62cc5aaae94cb0660facb91fc253a70c37c71"}
Nov 03 19:56:45.054540 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.054468    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"
Nov 03 19:56:45.075386 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.075318    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6" podStartSLOduration=1.075301472 podStartE2EDuration="1.075301472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:45.073666806 +0000 UTC m=+1157.228569225" watchObservedRunningTime="2026-11-03 19:56:45.075301472 +0000 UTC m=+1157.230203893"
Nov 03 19:56:45.225271 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:56:45.225236    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:56:45.374931 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.374892    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:45.743644 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.743608    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:45.772005 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.771972    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"]
Nov 03 19:56:45.776081 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.776063    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"
Nov 03 19:56:45.795152 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.795124    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"]
Nov 03 19:56:45.827760 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.827730    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9a6d4b\""
Nov 03 19:56:45.834696 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.834666    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd941a0e-4b03-4545-aab6-076a226a708b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72\" (UID: \"fd941a0e-4b03-4545-aab6-076a226a708b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"
Nov 03 19:56:45.834846 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.834802    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55knx\" (UniqueName: \"kubernetes.io/projected/fd941a0e-4b03-4545-aab6-076a226a708b-kube-api-access-55knx\") pod \"payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72\" (UID: \"fd941a0e-4b03-4545-aab6-076a226a708b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"
Nov 03 19:56:45.965508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.964630    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd941a0e-4b03-4545-aab6-076a226a708b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72\" (UID: \"fd941a0e-4b03-4545-aab6-076a226a708b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"
Nov 03 19:56:45.965508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.964742    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55knx\" (UniqueName: \"kubernetes.io/projected/fd941a0e-4b03-4545-aab6-076a226a708b-kube-api-access-55knx\") pod \"payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72\" (UID: \"fd941a0e-4b03-4545-aab6-076a226a708b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"
Nov 03 19:56:45.969503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:45.965985    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd941a0e-4b03-4545-aab6-076a226a708b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72\" (UID: \"fd941a0e-4b03-4545-aab6-076a226a708b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"
Nov 03 19:56:46.018551 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.014139    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"]
Nov 03 19:56:46.036501 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.027369    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55knx\" (UniqueName: \"kubernetes.io/projected/fd941a0e-4b03-4545-aab6-076a226a708b-kube-api-access-55knx\") pod \"payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72\" (UID: \"fd941a0e-4b03-4545-aab6-076a226a708b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"
Nov 03 19:56:46.080507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.067861    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:46.080507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.068886    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"
Nov 03 19:56:46.080507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.079827    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9a6d4b-dockercfg-9lvdf\""
Nov 03 19:56:46.096507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.091144    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"]
Nov 03 19:56:46.096507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.091434    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"
Nov 03 19:56:46.167470 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.166927    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca6aff5f-f058-451f-90b2-49f02082e8f4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd\" (UID: \"ca6aff5f-f058-451f-90b2-49f02082e8f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"
Nov 03 19:56:46.167470 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.167217    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6sf\" (UniqueName: \"kubernetes.io/projected/ca6aff5f-f058-451f-90b2-49f02082e8f4-kube-api-access-qx6sf\") pod \"payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd\" (UID: \"ca6aff5f-f058-451f-90b2-49f02082e8f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"
Nov 03 19:56:46.285331 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.285231    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6sf\" (UniqueName: \"kubernetes.io/projected/ca6aff5f-f058-451f-90b2-49f02082e8f4-kube-api-access-qx6sf\") pod \"payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd\" (UID: \"ca6aff5f-f058-451f-90b2-49f02082e8f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"
Nov 03 19:56:46.286092 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.286062    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca6aff5f-f058-451f-90b2-49f02082e8f4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd\" (UID: \"ca6aff5f-f058-451f-90b2-49f02082e8f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"
Nov 03 19:56:46.288092 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.288071    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca6aff5f-f058-451f-90b2-49f02082e8f4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd\" (UID: \"ca6aff5f-f058-451f-90b2-49f02082e8f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"
Nov 03 19:56:46.298964 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.298841    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6sf\" (UniqueName: \"kubernetes.io/projected/ca6aff5f-f058-451f-90b2-49f02082e8f4-kube-api-access-qx6sf\") pod \"payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd\" (UID: \"ca6aff5f-f058-451f-90b2-49f02082e8f4\") " pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"
Nov 03 19:56:46.428061 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.428028    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"
Nov 03 19:56:46.467252 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.464047    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"]
Nov 03 19:56:46.469270 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:56:46.469230    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd941a0e_4b03_4545_aab6_076a226a708b.slice/crio-35c8521f96f8c7ce78bf69db436a198e5ebd4b2e3728f65819b1f39b4f0b9c10 WatchSource:0}: Error finding container 35c8521f96f8c7ce78bf69db436a198e5ebd4b2e3728f65819b1f39b4f0b9c10: Status 404 returned error can't find the container with id 35c8521f96f8c7ce78bf69db436a198e5ebd4b2e3728f65819b1f39b4f0b9c10
Nov 03 19:56:46.733361 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:56:46.731032    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6aff5f_f058_451f_90b2_49f02082e8f4.slice/crio-214ffc2d777b2aaa90e72003d4b8c044baedfa97971a5b2f5b968a97d6600f3f WatchSource:0}: Error finding container 214ffc2d777b2aaa90e72003d4b8c044baedfa97971a5b2f5b968a97d6600f3f: Status 404 returned error can't find the container with id 214ffc2d777b2aaa90e72003d4b8c044baedfa97971a5b2f5b968a97d6600f3f
Nov 03 19:56:46.733361 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.731664    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"]
Nov 03 19:56:46.754239 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:46.754212    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:47.127679 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:47.127584    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72" event={"ID":"fd941a0e-4b03-4545-aab6-076a226a708b","Type":"ContainerStarted","Data":"84380130eac78c4d1b432e3f53de52d895f96cbfed8aedbd04610db374f8f379"}
Nov 03 19:56:47.127679 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:47.127655    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72" event={"ID":"fd941a0e-4b03-4545-aab6-076a226a708b","Type":"ContainerStarted","Data":"35c8521f96f8c7ce78bf69db436a198e5ebd4b2e3728f65819b1f39b4f0b9c10"}
Nov 03 19:56:47.127876 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:47.127793    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"
Nov 03 19:56:47.129351 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:47.129325    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd" event={"ID":"ca6aff5f-f058-451f-90b2-49f02082e8f4","Type":"ContainerStarted","Data":"cfd8c92a19dc8e67b7f01db6d8fc4446a30a9cee852e723963289cbe8e946f3e"}
Nov 03 19:56:47.129476 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:47.129366    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd" event={"ID":"ca6aff5f-f058-451f-90b2-49f02082e8f4","Type":"ContainerStarted","Data":"214ffc2d777b2aaa90e72003d4b8c044baedfa97971a5b2f5b968a97d6600f3f"}
Nov 03 19:56:47.129476 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:47.129430    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"
Nov 03 19:56:47.149138 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:47.149083    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72" podStartSLOduration=2.14906821 podStartE2EDuration="2.14906821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:47.14576246 +0000 UTC m=+1159.300664879" watchObservedRunningTime="2026-11-03 19:56:47.14906821 +0000 UTC m=+1159.303970629"
Nov 03 19:56:47.166909 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:47.166853    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd" podStartSLOduration=2.1668398460000002 podStartE2EDuration="2.166839846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:47.164698401 +0000 UTC m=+1159.319600820" watchObservedRunningTime="2026-11-03 19:56:47.166839846 +0000 UTC m=+1159.321742265"
Nov 03 19:56:47.602260 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:47.602228    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:47.736985 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:47.736951    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:48.595240 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:48.595198    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:49.295922 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:49.295889    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:49.872405 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:49.872374    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:50.428233 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:50.428200    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:50.557656 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:50.557619    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:51.343299 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:51.343234    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:52.037739 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:52.037702    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:52.725610 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:52.725576    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:52.865934 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:52.865894    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:53.002145 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:53.002065    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:53.870270 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:53.870235    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:54.256692 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:54.256648    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:54.380127 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:54.380092    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:55.122466 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:55.122433    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:55.245089 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:55.245055    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:55.445236 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:55.445154    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:56.135171 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:56.135140    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"
Nov 03 19:56:56.652724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:56.652685    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:57.548022 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:57.547986    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:57.695567 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:57.695529    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:57.827562 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:57.827449    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:58.018204 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:58.018169    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:58.140251 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:58.140166    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"
Nov 03 19:56:58.140251 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:58.140245    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"
Nov 03 19:56:58.464564 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:58.460261    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:58.633108 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:58.633041    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:59.280360 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:59.280328    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:59.563150 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:59.563067    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:59.716041 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:59.716006    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:59.851226 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:56:59.850335    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:00.549506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:00.545273    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:01.150837 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:01.150807    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:01.498836 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:01.498797    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:02.591628 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:02.591587    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:03.288986 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:03.288862    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:03.536888 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:03.536854    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:03.679221 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:03.679124    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:04.223133 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:04.223097    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:04.769473 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:04.769437    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:04.840973 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:04.840941    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"]
Nov 03 19:57:04.846633 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:04.846607    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"
Nov 03 19:57:04.868537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:04.868507    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"]
Nov 03 19:57:04.981744 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:04.981712    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6a7dc9-dockercfg-6cmbr\""
Nov 03 19:57:04.983618 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:04.983352    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6a7dc9\""
Nov 03 19:57:04.983918 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:04.983529    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hzg4\" (UniqueName: \"kubernetes.io/projected/670fc60a-c585-4c83-a527-143c4c951f89-kube-api-access-8hzg4\") pod \"payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9\" (UID: \"670fc60a-c585-4c83-a527-143c4c951f89\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"
Nov 03 19:57:04.984137 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:04.984119    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670fc60a-c585-4c83-a527-143c4c951f89-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9\" (UID: \"670fc60a-c585-4c83-a527-143c4c951f89\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"
Nov 03 19:57:05.021558 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:05.021439    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:05.093626 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:05.093584    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hzg4\" (UniqueName: \"kubernetes.io/projected/670fc60a-c585-4c83-a527-143c4c951f89-kube-api-access-8hzg4\") pod \"payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9\" (UID: \"670fc60a-c585-4c83-a527-143c4c951f89\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"
Nov 03 19:57:05.093965 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:05.093935    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670fc60a-c585-4c83-a527-143c4c951f89-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9\" (UID: \"670fc60a-c585-4c83-a527-143c4c951f89\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"
Nov 03 19:57:05.096198 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:05.096148    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670fc60a-c585-4c83-a527-143c4c951f89-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9\" (UID: \"670fc60a-c585-4c83-a527-143c4c951f89\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"
Nov 03 19:57:05.109572 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:05.109540    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hzg4\" (UniqueName: \"kubernetes.io/projected/670fc60a-c585-4c83-a527-143c4c951f89-kube-api-access-8hzg4\") pod \"payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9\" (UID: \"670fc60a-c585-4c83-a527-143c4c951f89\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"
Nov 03 19:57:05.154832 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:05.154797    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:05.179776 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:05.179743    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"
Nov 03 19:57:05.484843 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:05.484815    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"]
Nov 03 19:57:05.860912 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:05.859358    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:06.178602 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:06.173982    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:06.368917 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:06.368875    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9" event={"ID":"670fc60a-c585-4c83-a527-143c4c951f89","Type":"ContainerStarted","Data":"1fa9faf33bc5c45ca29e7e3ca950e4c3802dffe794a1ab87e62b343c4a71a0fe"}
Nov 03 19:57:06.369119 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:06.368941    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9" event={"ID":"670fc60a-c585-4c83-a527-143c4c951f89","Type":"ContainerStarted","Data":"622fb61c729a79d6d03c87848eea52b7b27d35704ac8a22a5977b0f5f189aac8"}
Nov 03 19:57:06.369609 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:06.369587    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"
Nov 03 19:57:06.389153 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:06.389094    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9" podStartSLOduration=2.389081807 podStartE2EDuration="2.389081807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:06.387965602 +0000 UTC m=+1178.542868022" watchObservedRunningTime="2026-11-03 19:57:06.389081807 +0000 UTC m=+1178.543984225"
Nov 03 19:57:07.297810 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:07.297772    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:08.141512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:08.141387    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:08.287983 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:08.287950    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:08.428862 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:08.428785    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:08.665265 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:08.665226    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:09.025061 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:09.024957    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:09.521715 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:09.521669    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:09.842019 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:09.841921    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:10.384369 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:10.384321    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:12.017460 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:12.017430    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:12.228940 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:12.228910    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:12.907000 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:12.906964    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:13.360506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:13.355072    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:13.591371 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:13.587005    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"]
Nov 03 19:57:13.639013 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:13.625370    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:13.783017 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:13.782982    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:13.998179 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:13.998146    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:14.616519 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:14.613643    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:14.870761 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:14.870680    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"]
Nov 03 19:57:14.908359 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:14.908331    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"]
Nov 03 19:57:14.992529 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:14.992423    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:15.164661 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:15.164551    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:15.523189 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:57:15.523150    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:57:16.117408 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:16.113584    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:16.296373 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:16.296319    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:17.177204 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:17.177155    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:17.392074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:17.392036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:17.396146 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:17.396119    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"
Nov 03 19:57:17.848501 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:17.825201    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:18.147568 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:18.144752    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:18.596507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:18.595591    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6" podUID="529d2c30-6f5e-4f21-a959-66b2f5904229" containerName="payload-pre-processing" containerID="cri-o://5d3b9a422366b6eb79c6af1d86b132f9a60d029f2612f3058843f5b53f3180cc" gracePeriod=25
Nov 03 19:57:18.927214 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:18.927176    5659 generic.go:356] "Generic (PLEG): container finished" podID="529d2c30-6f5e-4f21-a959-66b2f5904229" containerID="5d3b9a422366b6eb79c6af1d86b132f9a60d029f2612f3058843f5b53f3180cc" exitCode=0
Nov 03 19:57:18.927402 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:18.927382    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6" event={"ID":"529d2c30-6f5e-4f21-a959-66b2f5904229","Type":"ContainerDied","Data":"5d3b9a422366b6eb79c6af1d86b132f9a60d029f2612f3058843f5b53f3180cc"}
Nov 03 19:57:19.099129 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.096528    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:19.206363 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.206337    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"
Nov 03 19:57:19.219611 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.219580    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529d2c30-6f5e-4f21-a959-66b2f5904229-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529d2c30-6f5e-4f21-a959-66b2f5904229-plugins-config-volume\") pod \"529d2c30-6f5e-4f21-a959-66b2f5904229\" (UID: \"529d2c30-6f5e-4f21-a959-66b2f5904229\") "
Nov 03 19:57:19.219869 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.219852    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529d2c30-6f5e-4f21-a959-66b2f5904229-kube-api-access-glj4c\" (UniqueName: \"kubernetes.io/projected/529d2c30-6f5e-4f21-a959-66b2f5904229-kube-api-access-glj4c\") pod \"529d2c30-6f5e-4f21-a959-66b2f5904229\" (UID: \"529d2c30-6f5e-4f21-a959-66b2f5904229\") "
Nov 03 19:57:19.219976 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.219949    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529d2c30-6f5e-4f21-a959-66b2f5904229-plugins-config-volume" pod "529d2c30-6f5e-4f21-a959-66b2f5904229" (UID: "529d2c30-6f5e-4f21-a959-66b2f5904229"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:19.220675 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.220625    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529d2c30-6f5e-4f21-a959-66b2f5904229-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:19.222194 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.222169    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529d2c30-6f5e-4f21-a959-66b2f5904229-kube-api-access-glj4c" pod "529d2c30-6f5e-4f21-a959-66b2f5904229" (UID: "529d2c30-6f5e-4f21-a959-66b2f5904229"). InnerVolumeSpecName "kube-api-access-glj4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:19.322006 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.321972    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glj4c\" (UniqueName: \"kubernetes.io/projected/529d2c30-6f5e-4f21-a959-66b2f5904229-kube-api-access-glj4c\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:19.864042 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.864005    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:19.875072 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.873624    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph" podUID="b951e973-55bb-4eb5-a882-8a0eef28572f" containerName="payload-processing" containerID="cri-o://83e61ef9001e7e9013d40d5a7879087b70e4d3b48bd88b878d4853824ceeac9d" gracePeriod=25
Nov 03 19:57:19.909348 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.909302    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds" podUID="1d7f216d-bf8d-4e20-811e-19cc85424099" containerName="payload-pre-processing" containerID="cri-o://036d29b83129106faf86cc2a0f573eb97e45166f477fdef8cce4d5fa21c2dc0f" gracePeriod=25
Nov 03 19:57:19.946516 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.946398    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"
Nov 03 19:57:19.946516 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.946435    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6" event={"ID":"529d2c30-6f5e-4f21-a959-66b2f5904229","Type":"ContainerDied","Data":"6f574d1c23a616391129fdf338a62cc5aaae94cb0660facb91fc253a70c37c71"}
Nov 03 19:57:19.946768 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:19.946527    5659 scope.go:122] "RemoveContainer" containerID="5d3b9a422366b6eb79c6af1d86b132f9a60d029f2612f3058843f5b53f3180cc"
Nov 03 19:57:20.043523 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.040005    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"]
Nov 03 19:57:20.075609 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.075497    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b3619-7f99685759-5tzq6"]
Nov 03 19:57:20.261240 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.261206    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"
Nov 03 19:57:20.304166 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.304135    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"
Nov 03 19:57:20.460232 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.460193    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b951e973-55bb-4eb5-a882-8a0eef28572f-kube-api-access-8kf2z\" (UniqueName: \"kubernetes.io/projected/b951e973-55bb-4eb5-a882-8a0eef28572f-kube-api-access-8kf2z\") pod \"b951e973-55bb-4eb5-a882-8a0eef28572f\" (UID: \"b951e973-55bb-4eb5-a882-8a0eef28572f\") "
Nov 03 19:57:20.460431 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.460298    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b951e973-55bb-4eb5-a882-8a0eef28572f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b951e973-55bb-4eb5-a882-8a0eef28572f-plugins-config-volume\") pod \"b951e973-55bb-4eb5-a882-8a0eef28572f\" (UID: \"b951e973-55bb-4eb5-a882-8a0eef28572f\") "
Nov 03 19:57:20.460431 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.460376    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7f216d-bf8d-4e20-811e-19cc85424099-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7f216d-bf8d-4e20-811e-19cc85424099-plugins-config-volume\") pod \"1d7f216d-bf8d-4e20-811e-19cc85424099\" (UID: \"1d7f216d-bf8d-4e20-811e-19cc85424099\") "
Nov 03 19:57:20.460431 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.460419    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7f216d-bf8d-4e20-811e-19cc85424099-kube-api-access-8z9dv\" (UniqueName: \"kubernetes.io/projected/1d7f216d-bf8d-4e20-811e-19cc85424099-kube-api-access-8z9dv\") pod \"1d7f216d-bf8d-4e20-811e-19cc85424099\" (UID: \"1d7f216d-bf8d-4e20-811e-19cc85424099\") "
Nov 03 19:57:20.461217 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.461005    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b951e973-55bb-4eb5-a882-8a0eef28572f-plugins-config-volume" pod "b951e973-55bb-4eb5-a882-8a0eef28572f" (UID: "b951e973-55bb-4eb5-a882-8a0eef28572f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:20.461217 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.461117    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b951e973-55bb-4eb5-a882-8a0eef28572f-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:20.461677 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.461605    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7f216d-bf8d-4e20-811e-19cc85424099-plugins-config-volume" pod "1d7f216d-bf8d-4e20-811e-19cc85424099" (UID: "1d7f216d-bf8d-4e20-811e-19cc85424099"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:20.463738 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.463712    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7f216d-bf8d-4e20-811e-19cc85424099-kube-api-access-8z9dv" pod "1d7f216d-bf8d-4e20-811e-19cc85424099" (UID: "1d7f216d-bf8d-4e20-811e-19cc85424099"). InnerVolumeSpecName "kube-api-access-8z9dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:20.463993 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.463971    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b951e973-55bb-4eb5-a882-8a0eef28572f-kube-api-access-8kf2z" pod "b951e973-55bb-4eb5-a882-8a0eef28572f" (UID: "b951e973-55bb-4eb5-a882-8a0eef28572f"). InnerVolumeSpecName "kube-api-access-8kf2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:20.563504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.562602    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kf2z\" (UniqueName: \"kubernetes.io/projected/b951e973-55bb-4eb5-a882-8a0eef28572f-kube-api-access-8kf2z\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:20.563504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.562634    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7f216d-bf8d-4e20-811e-19cc85424099-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:20.563504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.562644    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z9dv\" (UniqueName: \"kubernetes.io/projected/1d7f216d-bf8d-4e20-811e-19cc85424099-kube-api-access-8z9dv\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:20.658507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.653290    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:20.968506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.963309    5659 generic.go:356] "Generic (PLEG): container finished" podID="1d7f216d-bf8d-4e20-811e-19cc85424099" containerID="036d29b83129106faf86cc2a0f573eb97e45166f477fdef8cce4d5fa21c2dc0f" exitCode=0
Nov 03 19:57:20.968506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.963381    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds" event={"ID":"1d7f216d-bf8d-4e20-811e-19cc85424099","Type":"ContainerDied","Data":"036d29b83129106faf86cc2a0f573eb97e45166f477fdef8cce4d5fa21c2dc0f"}
Nov 03 19:57:20.968506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.963432    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds" event={"ID":"1d7f216d-bf8d-4e20-811e-19cc85424099","Type":"ContainerDied","Data":"f6320929b63176396601b7087ede7a54d1472b1e0237f3473bf73f522012b7dd"}
Nov 03 19:57:20.968506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.963463    5659 scope.go:122] "RemoveContainer" containerID="036d29b83129106faf86cc2a0f573eb97e45166f477fdef8cce4d5fa21c2dc0f"
Nov 03 19:57:20.968506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.963783    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"
Nov 03 19:57:20.969528 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.969313    5659 generic.go:356] "Generic (PLEG): container finished" podID="b951e973-55bb-4eb5-a882-8a0eef28572f" containerID="83e61ef9001e7e9013d40d5a7879087b70e4d3b48bd88b878d4853824ceeac9d" exitCode=0
Nov 03 19:57:20.969528 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.969458    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph" event={"ID":"b951e973-55bb-4eb5-a882-8a0eef28572f","Type":"ContainerDied","Data":"83e61ef9001e7e9013d40d5a7879087b70e4d3b48bd88b878d4853824ceeac9d"}
Nov 03 19:57:20.969528 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.969522    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph" event={"ID":"b951e973-55bb-4eb5-a882-8a0eef28572f","Type":"ContainerDied","Data":"46a8ef63a0384a2abfd2cfcd3a919d0ef2cf11221f4904fd8d9943c6c0c31ade"}
Nov 03 19:57:20.969700 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.969607    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"
Nov 03 19:57:20.985736 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.985708    5659 scope.go:122] "RemoveContainer" containerID="036d29b83129106faf86cc2a0f573eb97e45166f477fdef8cce4d5fa21c2dc0f"
Nov 03 19:57:20.986079 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:57:20.986050    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036d29b83129106faf86cc2a0f573eb97e45166f477fdef8cce4d5fa21c2dc0f\": container with ID starting with 036d29b83129106faf86cc2a0f573eb97e45166f477fdef8cce4d5fa21c2dc0f not found: ID does not exist" containerID="036d29b83129106faf86cc2a0f573eb97e45166f477fdef8cce4d5fa21c2dc0f"
Nov 03 19:57:20.986172 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.986096    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036d29b83129106faf86cc2a0f573eb97e45166f477fdef8cce4d5fa21c2dc0f"} err="failed to get container status \"036d29b83129106faf86cc2a0f573eb97e45166f477fdef8cce4d5fa21c2dc0f\": rpc error: code = NotFound desc = could not find container \"036d29b83129106faf86cc2a0f573eb97e45166f477fdef8cce4d5fa21c2dc0f\": container with ID starting with 036d29b83129106faf86cc2a0f573eb97e45166f477fdef8cce4d5fa21c2dc0f not found: ID does not exist"
Nov 03 19:57:20.986172 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.986123    5659 scope.go:122] "RemoveContainer" containerID="83e61ef9001e7e9013d40d5a7879087b70e4d3b48bd88b878d4853824ceeac9d"
Nov 03 19:57:20.999924 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:20.999857    5659 scope.go:122] "RemoveContainer" containerID="83e61ef9001e7e9013d40d5a7879087b70e4d3b48bd88b878d4853824ceeac9d"
Nov 03 19:57:21.000237 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:57:21.000214    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e61ef9001e7e9013d40d5a7879087b70e4d3b48bd88b878d4853824ceeac9d\": container with ID starting with 83e61ef9001e7e9013d40d5a7879087b70e4d3b48bd88b878d4853824ceeac9d not found: ID does not exist" containerID="83e61ef9001e7e9013d40d5a7879087b70e4d3b48bd88b878d4853824ceeac9d"
Nov 03 19:57:21.000318 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:21.000250    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e61ef9001e7e9013d40d5a7879087b70e4d3b48bd88b878d4853824ceeac9d"} err="failed to get container status \"83e61ef9001e7e9013d40d5a7879087b70e4d3b48bd88b878d4853824ceeac9d\": rpc error: code = NotFound desc = could not find container \"83e61ef9001e7e9013d40d5a7879087b70e4d3b48bd88b878d4853824ceeac9d\": container with ID starting with 83e61ef9001e7e9013d40d5a7879087b70e4d3b48bd88b878d4853824ceeac9d not found: ID does not exist"
Nov 03 19:57:21.008946 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:21.008917    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"]
Nov 03 19:57:21.014609 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:21.014584    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b0f9d6-7bb8df46d6-gp5ph"]
Nov 03 19:57:21.027552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:21.027523    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"]
Nov 03 19:57:21.035861 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:21.035832    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b0f9d6-84b757d4c9-qh7ds"]
Nov 03 19:57:21.392407 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:21.392374    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:21.913005 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:21.912968    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:21.974466 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:21.974418    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7f216d-bf8d-4e20-811e-19cc85424099" path="/var/lib/kubelet/pods/1d7f216d-bf8d-4e20-811e-19cc85424099/volumes"
Nov 03 19:57:21.974971 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:21.974925    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529d2c30-6f5e-4f21-a959-66b2f5904229" path="/var/lib/kubelet/pods/529d2c30-6f5e-4f21-a959-66b2f5904229/volumes"
Nov 03 19:57:21.975341 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:21.975316    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b951e973-55bb-4eb5-a882-8a0eef28572f" path="/var/lib/kubelet/pods/b951e973-55bb-4eb5-a882-8a0eef28572f/volumes"
Nov 03 19:57:22.070069 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:22.070029    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:22.320405 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:22.320365    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"]
Nov 03 19:57:22.438542 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:22.438506    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:22.576439 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:22.576339    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:23.299235 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:23.299200    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:23.603904 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:23.603821    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:24.239032 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:24.239005    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:24.388170 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:24.388082    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:24.694991 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:24.694893    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:24.830660 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:24.830627    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:24.919869 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:24.919829    5659 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:57:24.920117 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:24.920089    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:57:25.018467 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.018434    5659 generic.go:356] "Generic (PLEG): container finished" podID="561ecf04-5f23-49de-8881-639fc816fa81" containerID="a6d4f53c0f90986fe14e3c56f54098237bbb171256da82b36b7f36ca7323de5e" exitCode=0
Nov 03 19:57:25.018674 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.018512    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" event={"ID":"561ecf04-5f23-49de-8881-639fc816fa81","Type":"ContainerDied","Data":"a6d4f53c0f90986fe14e3c56f54098237bbb171256da82b36b7f36ca7323de5e"}
Nov 03 19:57:25.019027 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.019005    5659 scope.go:122] "RemoveContainer" containerID="a6d4f53c0f90986fe14e3c56f54098237bbb171256da82b36b7f36ca7323de5e"
Nov 03 19:57:25.020223 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.020204    5659 generic.go:356] "Generic (PLEG): container finished" podID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerID="4fa08e00988778f86683bc137aa8a81af9a8e5dad6f8cbcac1803a13915ba093" exitCode=0
Nov 03 19:57:25.020312 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.020272    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" event={"ID":"a49ffc14-16a0-495d-954f-a524e0d4a81a","Type":"ContainerDied","Data":"4fa08e00988778f86683bc137aa8a81af9a8e5dad6f8cbcac1803a13915ba093"}
Nov 03 19:57:25.020758 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.020740    5659 scope.go:122] "RemoveContainer" containerID="4fa08e00988778f86683bc137aa8a81af9a8e5dad6f8cbcac1803a13915ba093"
Nov 03 19:57:25.020972 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.020941    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:57:25.021043 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.021025    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:57:25.021110 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.021093    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:57:25.022566 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.022478    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:57:25.022651 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.022615    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:57:25.022695 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.022662    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:57:25.442460 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.442364    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:25.591645 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.591605    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:25.908539 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:25.905409    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:26.114509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:26.110639    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" event={"ID":"561ecf04-5f23-49de-8881-639fc816fa81","Type":"ContainerStarted","Data":"b27670d4c7a9eb0b406f95b19cd9cf735a435ae5733950ac2713f8bff8d1aa63"}
Nov 03 19:57:26.123505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:26.115008    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" event={"ID":"a49ffc14-16a0-495d-954f-a524e0d4a81a","Type":"ContainerStarted","Data":"ba87f529ff35b90e8c98f1fe22095b1a8c6b3a55b720d255e56129b7c0f2fd10"}
Nov 03 19:57:26.551515 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:26.550934    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:57:26.551515 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:26.551064    5659 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:57:26.551515 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:26.551108    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:57:26.624806 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:26.624771    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:57:26.626169 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:26.626137    5659 patch_prober.go:29] interesting pod/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:57:26.626340 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:26.626235    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" podUID="561ecf04-5f23-49de-8881-639fc816fa81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 19:57:26.922331 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:26.922251    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:26.945526 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:26.942611    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:57:27.091848 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:27.091807    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:27.321509 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:27.321161    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq" podUID="339756bb-9cd5-4a08-a142-93ea4affded0" containerName="payload-pre-processing" containerID="cri-o://8ffe099e5ee0671eb48b8f918890f5fdcba9476aaebe58a1aaa9e6141bb8912e" gracePeriod=25
Nov 03 19:57:27.562031 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:27.561166    5659 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:57:27.562031 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:27.561224    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:57:27.639589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:27.633894    5659 patch_prober.go:29] interesting pod/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:57:27.639589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:27.633953    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" podUID="561ecf04-5f23-49de-8881-639fc816fa81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 19:57:27.655857 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:27.655798    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:28.249568 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.227797    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:28.511699 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.508701    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:28.521326 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.521299    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"
Nov 03 19:57:28.525121 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.525093    5659 generic.go:356] "Generic (PLEG): container finished" podID="339756bb-9cd5-4a08-a142-93ea4affded0" containerID="8ffe099e5ee0671eb48b8f918890f5fdcba9476aaebe58a1aaa9e6141bb8912e" exitCode=0
Nov 03 19:57:28.526624 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.526604    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"
Nov 03 19:57:28.526920 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.526887    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq" event={"ID":"339756bb-9cd5-4a08-a142-93ea4affded0","Type":"ContainerDied","Data":"8ffe099e5ee0671eb48b8f918890f5fdcba9476aaebe58a1aaa9e6141bb8912e"}
Nov 03 19:57:28.527083 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.527070    5659 scope.go:122] "RemoveContainer" containerID="8ffe099e5ee0671eb48b8f918890f5fdcba9476aaebe58a1aaa9e6141bb8912e"
Nov 03 19:57:28.605512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.594338    5659 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:57:28.605512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.594429    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:57:28.614749 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.612693    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/339756bb-9cd5-4a08-a142-93ea4affded0-kube-api-access-576qj\" (UniqueName: \"kubernetes.io/projected/339756bb-9cd5-4a08-a142-93ea4affded0-kube-api-access-576qj\") pod \"339756bb-9cd5-4a08-a142-93ea4affded0\" (UID: \"339756bb-9cd5-4a08-a142-93ea4affded0\") "
Nov 03 19:57:28.614749 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.612738    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/339756bb-9cd5-4a08-a142-93ea4affded0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339756bb-9cd5-4a08-a142-93ea4affded0-plugins-config-volume\") pod \"339756bb-9cd5-4a08-a142-93ea4affded0\" (UID: \"339756bb-9cd5-4a08-a142-93ea4affded0\") "
Nov 03 19:57:28.614749 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.613879    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339756bb-9cd5-4a08-a142-93ea4affded0-plugins-config-volume" pod "339756bb-9cd5-4a08-a142-93ea4affded0" (UID: "339756bb-9cd5-4a08-a142-93ea4affded0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:28.618513 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.616369    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339756bb-9cd5-4a08-a142-93ea4affded0-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:28.622505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.621992    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339756bb-9cd5-4a08-a142-93ea4affded0-kube-api-access-576qj" pod "339756bb-9cd5-4a08-a142-93ea4affded0" (UID: "339756bb-9cd5-4a08-a142-93ea4affded0"). InnerVolumeSpecName "kube-api-access-576qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:28.631314 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.631288    5659 patch_prober.go:29] interesting pod/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:57:28.631565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.631535    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" podUID="561ecf04-5f23-49de-8881-639fc816fa81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 19:57:28.715519 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:57:28.714882    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffe099e5ee0671eb48b8f918890f5fdcba9476aaebe58a1aaa9e6141bb8912e\": container with ID starting with 8ffe099e5ee0671eb48b8f918890f5fdcba9476aaebe58a1aaa9e6141bb8912e not found: ID does not exist" containerID="8ffe099e5ee0671eb48b8f918890f5fdcba9476aaebe58a1aaa9e6141bb8912e"
Nov 03 19:57:28.715519 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:57:28.714935    5659 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"8ffe099e5ee0671eb48b8f918890f5fdcba9476aaebe58a1aaa9e6141bb8912e\": container with ID starting with 8ffe099e5ee0671eb48b8f918890f5fdcba9476aaebe58a1aaa9e6141bb8912e not found: ID does not exist"
Nov 03 19:57:28.715519 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.715113    5659 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"8ffe099e5ee0671eb48b8f918890f5fdcba9476aaebe58a1aaa9e6141bb8912e\": container with ID starting with 8ffe099e5ee0671eb48b8f918890f5fdcba9476aaebe58a1aaa9e6141bb8912e not found: ID does not exist"
Nov 03 19:57:28.718709 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.718677    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-576qj\" (UniqueName: \"kubernetes.io/projected/339756bb-9cd5-4a08-a142-93ea4affded0-kube-api-access-576qj\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:28.900622 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.900569    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"]
Nov 03 19:57:28.904677 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:28.901180    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34f6c-5f9ff8598c-trbdq"]
Nov 03 19:57:29.183562 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:29.179228    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:29.564765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:29.560704    5659 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:57:29.564765 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:29.560831    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:57:29.634506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:29.628391    5659 patch_prober.go:29] interesting pod/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:57:29.634506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:29.628506    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" podUID="561ecf04-5f23-49de-8881-639fc816fa81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 19:57:29.704510 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:29.703145    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:29.993793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:29.993369    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="339756bb-9cd5-4a08-a142-93ea4affded0" path="/var/lib/kubelet/pods/339756bb-9cd5-4a08-a142-93ea4affded0/volumes"
Nov 03 19:57:30.568358 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:30.567892    5659 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:57:30.568358 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:30.567964    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:57:30.625585 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:30.625117    5659 patch_prober.go:29] interesting pod/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:57:30.625585 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:30.625182    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" podUID="561ecf04-5f23-49de-8881-639fc816fa81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 19:57:30.899560 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:30.894756    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:31.123565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:31.123524    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:31.512040 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:31.511758    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:31.554508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:31.552938    5659 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:57:31.554508 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:31.553013    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 19:57:31.626205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:31.626167    5659 patch_prober.go:29] interesting pod/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:57:31.626375 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:31.626229    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" podUID="561ecf04-5f23-49de-8881-639fc816fa81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 19:57:32.009176 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:32.008953    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:32.556079 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:32.556048    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:57:32.556413 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:32.556386    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:57:32.557296 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:32.557276    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 19:57:32.630551 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:32.630522    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:57:32.631888 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:32.631863    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:57:32.711455 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:32.711424    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:32.918362 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:32.918278    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:33.990544 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:33.987800    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:34.626930 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:34.626902    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:34.827818 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:34.827783    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:35.429755 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:35.429725    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:35.658202 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:35.658167    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:36.303629 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:36.303595    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:37.049730 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:37.049694    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:37.770811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:37.770773    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:38.710332 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:38.710287    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:38.751729 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:38.751697    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"]
Nov 03 19:57:38.771626 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:38.771593    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"]
Nov 03 19:57:38.919086 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:38.919056    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:39.082048 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:39.081961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:39.428387 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:39.428305    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:40.198702 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:40.198624    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:40.364063 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:40.363985    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:40.883552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:40.883449    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:41.094302 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:41.094267    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:41.914442 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:41.914286    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:42.681760 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:42.681688    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:43.644787 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:43.644743    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:43.753683 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:43.753584    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd" podUID="ca6aff5f-f058-451f-90b2-49f02082e8f4" containerName="payload-processing" containerID="cri-o://cfd8c92a19dc8e67b7f01db6d8fc4446a30a9cee852e723963289cbe8e946f3e" gracePeriod=25
Nov 03 19:57:43.778502 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:43.772589    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72" podUID="fd941a0e-4b03-4545-aab6-076a226a708b" containerName="payload-pre-processing" containerID="cri-o://84380130eac78c4d1b432e3f53de52d895f96cbfed8aedbd04610db374f8f379" gracePeriod=25
Nov 03 19:57:43.865809 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:43.865775    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:43.932323 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:43.931955    5659 generic.go:356] "Generic (PLEG): container finished" podID="ca6aff5f-f058-451f-90b2-49f02082e8f4" containerID="cfd8c92a19dc8e67b7f01db6d8fc4446a30a9cee852e723963289cbe8e946f3e" exitCode=0
Nov 03 19:57:43.932323 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:43.932194    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd" event={"ID":"ca6aff5f-f058-451f-90b2-49f02082e8f4","Type":"ContainerDied","Data":"cfd8c92a19dc8e67b7f01db6d8fc4446a30a9cee852e723963289cbe8e946f3e"}
Nov 03 19:57:43.934457 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:43.934363    5659 generic.go:356] "Generic (PLEG): container finished" podID="fd941a0e-4b03-4545-aab6-076a226a708b" containerID="84380130eac78c4d1b432e3f53de52d895f96cbfed8aedbd04610db374f8f379" exitCode=0
Nov 03 19:57:43.934457 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:43.934408    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72" event={"ID":"fd941a0e-4b03-4545-aab6-076a226a708b","Type":"ContainerDied","Data":"84380130eac78c4d1b432e3f53de52d895f96cbfed8aedbd04610db374f8f379"}
Nov 03 19:57:44.008536 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.008509    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:44.106671 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.106643    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"
Nov 03 19:57:44.109863 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.109842    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"
Nov 03 19:57:44.223877 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.223846    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca6aff5f-f058-451f-90b2-49f02082e8f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca6aff5f-f058-451f-90b2-49f02082e8f4-plugins-config-volume\") pod \"ca6aff5f-f058-451f-90b2-49f02082e8f4\" (UID: \"ca6aff5f-f058-451f-90b2-49f02082e8f4\") "
Nov 03 19:57:44.224044 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.223904    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca6aff5f-f058-451f-90b2-49f02082e8f4-kube-api-access-qx6sf\" (UniqueName: \"kubernetes.io/projected/ca6aff5f-f058-451f-90b2-49f02082e8f4-kube-api-access-qx6sf\") pod \"ca6aff5f-f058-451f-90b2-49f02082e8f4\" (UID: \"ca6aff5f-f058-451f-90b2-49f02082e8f4\") "
Nov 03 19:57:44.224109 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.224087    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd941a0e-4b03-4545-aab6-076a226a708b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd941a0e-4b03-4545-aab6-076a226a708b-plugins-config-volume\") pod \"fd941a0e-4b03-4545-aab6-076a226a708b\" (UID: \"fd941a0e-4b03-4545-aab6-076a226a708b\") "
Nov 03 19:57:44.224145 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.224135    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd941a0e-4b03-4545-aab6-076a226a708b-kube-api-access-55knx\" (UniqueName: \"kubernetes.io/projected/fd941a0e-4b03-4545-aab6-076a226a708b-kube-api-access-55knx\") pod \"fd941a0e-4b03-4545-aab6-076a226a708b\" (UID: \"fd941a0e-4b03-4545-aab6-076a226a708b\") "
Nov 03 19:57:44.224428 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.224287    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6aff5f-f058-451f-90b2-49f02082e8f4-plugins-config-volume" pod "ca6aff5f-f058-451f-90b2-49f02082e8f4" (UID: "ca6aff5f-f058-451f-90b2-49f02082e8f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:44.224544 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.224459    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd941a0e-4b03-4545-aab6-076a226a708b-plugins-config-volume" pod "fd941a0e-4b03-4545-aab6-076a226a708b" (UID: "fd941a0e-4b03-4545-aab6-076a226a708b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:44.224855 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.224826    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca6aff5f-f058-451f-90b2-49f02082e8f4-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:44.224855 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.224848    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd941a0e-4b03-4545-aab6-076a226a708b-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:44.226234 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.226212    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6aff5f-f058-451f-90b2-49f02082e8f4-kube-api-access-qx6sf" pod "ca6aff5f-f058-451f-90b2-49f02082e8f4" (UID: "ca6aff5f-f058-451f-90b2-49f02082e8f4"). InnerVolumeSpecName "kube-api-access-qx6sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:44.226321 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.226235    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd941a0e-4b03-4545-aab6-076a226a708b-kube-api-access-55knx" pod "fd941a0e-4b03-4545-aab6-076a226a708b" (UID: "fd941a0e-4b03-4545-aab6-076a226a708b"). InnerVolumeSpecName "kube-api-access-55knx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:44.326426 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.326383    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx6sf\" (UniqueName: \"kubernetes.io/projected/ca6aff5f-f058-451f-90b2-49f02082e8f4-kube-api-access-qx6sf\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:44.326426 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.326424    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55knx\" (UniqueName: \"kubernetes.io/projected/fd941a0e-4b03-4545-aab6-076a226a708b-kube-api-access-55knx\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:44.763552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.763518    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:44.945400 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.945364    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"
Nov 03 19:57:44.945662 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.945391    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72" event={"ID":"fd941a0e-4b03-4545-aab6-076a226a708b","Type":"ContainerDied","Data":"35c8521f96f8c7ce78bf69db436a198e5ebd4b2e3728f65819b1f39b4f0b9c10"}
Nov 03 19:57:44.945662 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.945463    5659 scope.go:122] "RemoveContainer" containerID="84380130eac78c4d1b432e3f53de52d895f96cbfed8aedbd04610db374f8f379"
Nov 03 19:57:44.947964 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.947936    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd" event={"ID":"ca6aff5f-f058-451f-90b2-49f02082e8f4","Type":"ContainerDied","Data":"214ffc2d777b2aaa90e72003d4b8c044baedfa97971a5b2f5b968a97d6600f3f"}
Nov 03 19:57:44.948123 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.948036    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"
Nov 03 19:57:44.961659 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.961637    5659 scope.go:122] "RemoveContainer" containerID="cfd8c92a19dc8e67b7f01db6d8fc4446a30a9cee852e723963289cbe8e946f3e"
Nov 03 19:57:44.984632 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:44.984468    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"]
Nov 03 19:57:45.001456 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.001420    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9a6d4b-69cd6fcb47-lls72"]
Nov 03 19:57:45.014957 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.014880    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"]
Nov 03 19:57:45.019182 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.019157    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9a6d4b-6c567ddb8d-s97rd"]
Nov 03 19:57:45.220381 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:57:45.220343    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:57:45.313203 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.313107    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:45.419589 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.419549    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"]
Nov 03 19:57:45.420354 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420337    5659 cpu_manager.go:414] "containerMap: removing container" podUID="1d7f216d-bf8d-4e20-811e-19cc85424099" containerName="payload-pre-processing"
Nov 03 19:57:45.420427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420356    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7f216d-bf8d-4e20-811e-19cc85424099" containerName="payload-pre-processing"
Nov 03 19:57:45.420427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420384    5659 cpu_manager.go:414] "containerMap: removing container" podUID="fd941a0e-4b03-4545-aab6-076a226a708b" containerName="payload-pre-processing"
Nov 03 19:57:45.420427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420393    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd941a0e-4b03-4545-aab6-076a226a708b" containerName="payload-pre-processing"
Nov 03 19:57:45.420427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420407    5659 cpu_manager.go:414] "containerMap: removing container" podUID="339756bb-9cd5-4a08-a142-93ea4affded0" containerName="payload-pre-processing"
Nov 03 19:57:45.420427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420415    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339756bb-9cd5-4a08-a142-93ea4affded0" containerName="payload-pre-processing"
Nov 03 19:57:45.420601 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420433    5659 cpu_manager.go:414] "containerMap: removing container" podUID="ca6aff5f-f058-451f-90b2-49f02082e8f4" containerName="payload-processing"
Nov 03 19:57:45.420601 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420440    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6aff5f-f058-451f-90b2-49f02082e8f4" containerName="payload-processing"
Nov 03 19:57:45.420601 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420453    5659 cpu_manager.go:414] "containerMap: removing container" podUID="529d2c30-6f5e-4f21-a959-66b2f5904229" containerName="payload-pre-processing"
Nov 03 19:57:45.420601 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420461    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529d2c30-6f5e-4f21-a959-66b2f5904229" containerName="payload-pre-processing"
Nov 03 19:57:45.420601 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420494    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b951e973-55bb-4eb5-a882-8a0eef28572f" containerName="payload-processing"
Nov 03 19:57:45.420601 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420502    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b951e973-55bb-4eb5-a882-8a0eef28572f" containerName="payload-processing"
Nov 03 19:57:45.420769 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420619    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7f216d-bf8d-4e20-811e-19cc85424099" containerName="payload-pre-processing"
Nov 03 19:57:45.420769 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420632    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="339756bb-9cd5-4a08-a142-93ea4affded0" containerName="payload-pre-processing"
Nov 03 19:57:45.420769 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420642    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd941a0e-4b03-4545-aab6-076a226a708b" containerName="payload-pre-processing"
Nov 03 19:57:45.420769 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420654    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6aff5f-f058-451f-90b2-49f02082e8f4" containerName="payload-processing"
Nov 03 19:57:45.420769 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420664    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="529d2c30-6f5e-4f21-a959-66b2f5904229" containerName="payload-pre-processing"
Nov 03 19:57:45.420769 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.420674    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b951e973-55bb-4eb5-a882-8a0eef28572f" containerName="payload-processing"
Nov 03 19:57:45.425996 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.425975    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"
Nov 03 19:57:45.437901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.437875    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-29f633\""
Nov 03 19:57:45.438027 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.437973    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-29f633-dockercfg-np8s5\""
Nov 03 19:57:45.443745 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.443671    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"]
Nov 03 19:57:45.458250 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.458212    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"]
Nov 03 19:57:45.504335 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.504296    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:45.547422 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.545949    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5dnc\" (UniqueName: \"kubernetes.io/projected/30199a75-1c40-43e5-9c62-a279c19890d4-kube-api-access-q5dnc\") pod \"payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk\" (UID: \"30199a75-1c40-43e5-9c62-a279c19890d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"
Nov 03 19:57:45.547422 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.546063    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30199a75-1c40-43e5-9c62-a279c19890d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk\" (UID: \"30199a75-1c40-43e5-9c62-a279c19890d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"
Nov 03 19:57:45.658554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.650186    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5dnc\" (UniqueName: \"kubernetes.io/projected/30199a75-1c40-43e5-9c62-a279c19890d4-kube-api-access-q5dnc\") pod \"payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk\" (UID: \"30199a75-1c40-43e5-9c62-a279c19890d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"
Nov 03 19:57:45.658554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.650307    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30199a75-1c40-43e5-9c62-a279c19890d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk\" (UID: \"30199a75-1c40-43e5-9c62-a279c19890d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"
Nov 03 19:57:45.658554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.651318    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30199a75-1c40-43e5-9c62-a279c19890d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk\" (UID: \"30199a75-1c40-43e5-9c62-a279c19890d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"
Nov 03 19:57:45.702617 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.695743    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"]
Nov 03 19:57:45.702617 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.695903    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"
Nov 03 19:57:45.754329 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.754296    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbac47c-533a-4e63-b65b-c87a2a6eee14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg\" (UID: \"6dbac47c-533a-4e63-b65b-c87a2a6eee14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"
Nov 03 19:57:45.754683 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.754656    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fltj5\" (UniqueName: \"kubernetes.io/projected/6dbac47c-533a-4e63-b65b-c87a2a6eee14-kube-api-access-fltj5\") pod \"payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg\" (UID: \"6dbac47c-533a-4e63-b65b-c87a2a6eee14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"
Nov 03 19:57:45.806066 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.805969    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5dnc\" (UniqueName: \"kubernetes.io/projected/30199a75-1c40-43e5-9c62-a279c19890d4-kube-api-access-q5dnc\") pod \"payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk\" (UID: \"30199a75-1c40-43e5-9c62-a279c19890d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"
Nov 03 19:57:45.856168 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.856125    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbac47c-533a-4e63-b65b-c87a2a6eee14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg\" (UID: \"6dbac47c-533a-4e63-b65b-c87a2a6eee14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"
Nov 03 19:57:45.856358 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.856306    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fltj5\" (UniqueName: \"kubernetes.io/projected/6dbac47c-533a-4e63-b65b-c87a2a6eee14-kube-api-access-fltj5\") pod \"payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg\" (UID: \"6dbac47c-533a-4e63-b65b-c87a2a6eee14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"
Nov 03 19:57:45.857125 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.857093    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbac47c-533a-4e63-b65b-c87a2a6eee14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg\" (UID: \"6dbac47c-533a-4e63-b65b-c87a2a6eee14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"
Nov 03 19:57:45.874151 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.874119    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fltj5\" (UniqueName: \"kubernetes.io/projected/6dbac47c-533a-4e63-b65b-c87a2a6eee14-kube-api-access-fltj5\") pod \"payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg\" (UID: \"6dbac47c-533a-4e63-b65b-c87a2a6eee14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"
Nov 03 19:57:45.969596 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.969553    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca6aff5f-f058-451f-90b2-49f02082e8f4" path="/var/lib/kubelet/pods/ca6aff5f-f058-451f-90b2-49f02082e8f4/volumes"
Nov 03 19:57:45.970207 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:45.970188    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd941a0e-4b03-4545-aab6-076a226a708b" path="/var/lib/kubelet/pods/fd941a0e-4b03-4545-aab6-076a226a708b/volumes"
Nov 03 19:57:46.047955 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.047906    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"
Nov 03 19:57:46.066174 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.066143    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"
Nov 03 19:57:46.251403 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.251308    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:46.340942 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:57:46.340909    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30199a75_1c40_43e5_9c62_a279c19890d4.slice/crio-d16d3307fc0fcfcfc857127cd6b2ef02ee17417f0110fcb458d82fb90ebfc63c WatchSource:0}: Error finding container d16d3307fc0fcfcfc857127cd6b2ef02ee17417f0110fcb458d82fb90ebfc63c: Status 404 returned error can't find the container with id d16d3307fc0fcfcfc857127cd6b2ef02ee17417f0110fcb458d82fb90ebfc63c
Nov 03 19:57:46.351766 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.351717    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"]
Nov 03 19:57:46.412140 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.412053    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:46.472982 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.472935    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"]
Nov 03 19:57:46.944316 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.944213    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:46.971470 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.971208    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk" event={"ID":"30199a75-1c40-43e5-9c62-a279c19890d4","Type":"ContainerStarted","Data":"59be52ab2562757d6aca43c60814e6e1f658bbf83e6173105aaa6ae924e470b3"}
Nov 03 19:57:46.971470 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.971273    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk" event={"ID":"30199a75-1c40-43e5-9c62-a279c19890d4","Type":"ContainerStarted","Data":"d16d3307fc0fcfcfc857127cd6b2ef02ee17417f0110fcb458d82fb90ebfc63c"}
Nov 03 19:57:46.971470 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.971349    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"
Nov 03 19:57:46.973244 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.973202    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg" event={"ID":"6dbac47c-533a-4e63-b65b-c87a2a6eee14","Type":"ContainerStarted","Data":"5fe544f3c9310abaed43ec298bcaa449caa49e4761ea5ac8fef88daee316243c"}
Nov 03 19:57:46.973374 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.973251    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg" event={"ID":"6dbac47c-533a-4e63-b65b-c87a2a6eee14","Type":"ContainerStarted","Data":"c45a8f464670c83a3497d1dc1ffcc4cf0573c9f34265009af65717851cd99e10"}
Nov 03 19:57:46.973374 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.973319    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"
Nov 03 19:57:46.997987 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:46.997925    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk" podStartSLOduration=1.997907606 podStartE2EDuration="1.997907606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:46.997587743 +0000 UTC m=+1219.152490162" watchObservedRunningTime="2026-11-03 19:57:46.997907606 +0000 UTC m=+1219.152810026"
Nov 03 19:57:47.221046 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:47.221008    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:48.328710 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:48.328676    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:49.087727 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:49.087695    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:49.222878 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:49.222846    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:49.672503 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:49.668830    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:49.954649 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:49.953744    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg" podStartSLOduration=4.953724231 podStartE2EDuration="4.953724231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:47.05257328 +0000 UTC m=+1219.207475700" watchObservedRunningTime="2026-11-03 19:57:49.953724231 +0000 UTC m=+1222.108626651"
Nov 03 19:57:49.958397 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:49.958369    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"]
Nov 03 19:57:49.958880 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:49.958853    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" podUID="561ecf04-5f23-49de-8881-639fc816fa81" containerName="istio-proxy" containerID="cri-o://b27670d4c7a9eb0b406f95b19cd9cf735a435ae5733950ac2713f8bff8d1aa63" gracePeriod=30
Nov 03 19:57:50.021493 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:50.021459    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:50.341300 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:50.341207    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:50.481097 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:50.481066    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:51.360673 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:51.360644    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:51.950800 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:51.950770    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:52.600710 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.600633    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:52.610676 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.607604    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"]
Nov 03 19:57:52.623501 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.617668    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"]
Nov 03 19:57:52.623501 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.617779    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.623501 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.623424    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-1b1d04-openshift-default-dockercfg-gl687\""
Nov 03 19:57:52.707227 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.707167    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74da69e9-a168-4ae5-a882-daca94333466-istio-podinfo\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.707408 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.707240    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-socket\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.707408 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.707299    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-certs\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.707408 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.707388    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74da69e9-a168-4ae5-a882-daca94333466-istiod-ca-cert\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.707539 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.707422    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-istio-token\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.707539 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.707465    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-credential-socket\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.707539 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.707528    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-envoy\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.707634 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.707610    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5k5l\" (UniqueName: \"kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-kube-api-access-n5k5l\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.707666 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.707655    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-data\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.809606 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.809545    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-certs\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.811181 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.811155    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74da69e9-a168-4ae5-a882-daca94333466-istiod-ca-cert\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.811376 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.811358    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-istio-token\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.811629 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.811607    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-credential-socket\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.811781 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.811766    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-envoy\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.811954 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.811939    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5k5l\" (UniqueName: \"kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-kube-api-access-n5k5l\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.812180 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.812162    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-data\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.812392 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.812378    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74da69e9-a168-4ae5-a882-daca94333466-istio-podinfo\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.813929 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.813907    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-socket\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.814198 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.810243    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-certs\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.814316 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.813375    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74da69e9-a168-4ae5-a882-daca94333466-istiod-ca-cert\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.821661 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.821607    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-socket\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.822619 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.822592    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-data\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.823714 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.812797    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-credential-socket\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.826224 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.826140    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:52.840078 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.840047    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-envoy\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.843205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.843177    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74da69e9-a168-4ae5-a882-daca94333466-istio-podinfo\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.845593 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.845567    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-istio-token\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.858617 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.857890    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5k5l\" (UniqueName: \"kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-kube-api-access-n5k5l\") pod \"e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") " pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:52.968697 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:52.968657    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:53.159200 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:53.159105    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:53.300393 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:57:53.300363    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74da69e9_a168_4ae5_a882_daca94333466.slice/crio-72e87f9c7e4623787954d55e5886ac6f2a2c19de77c527ae5f9d1d9c5b130976 WatchSource:0}: Error finding container 72e87f9c7e4623787954d55e5886ac6f2a2c19de77c527ae5f9d1d9c5b130976: Status 404 returned error can't find the container with id 72e87f9c7e4623787954d55e5886ac6f2a2c19de77c527ae5f9d1d9c5b130976
Nov 03 19:57:53.301390 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:53.301365    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"]
Nov 03 19:57:53.303734 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:53.303701    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:57:53.303818 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:53.303765    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:57:53.303818 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:53.303797    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:57:54.025864 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:54.025827    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" event={"ID":"74da69e9-a168-4ae5-a882-daca94333466","Type":"ContainerStarted","Data":"96ee96b4fb32413f0f4a13240ff155b1f3e23752fb2bf428b899ad41fb553693"}
Nov 03 19:57:54.026247 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:54.025882    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" event={"ID":"74da69e9-a168-4ae5-a882-daca94333466","Type":"ContainerStarted","Data":"72e87f9c7e4623787954d55e5886ac6f2a2c19de77c527ae5f9d1d9c5b130976"}
Nov 03 19:57:54.059566 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:54.059505    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" podStartSLOduration=2.059471662 podStartE2EDuration="2.059471662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:54.057830727 +0000 UTC m=+1226.212733149" watchObservedRunningTime="2026-11-03 19:57:54.059471662 +0000 UTC m=+1226.214374082"
Nov 03 19:57:54.109786 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:54.109740    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:54.968924 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:54.968891    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:54.974732 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:54.974705    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:55.032672 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.032633    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:55.033650 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.033630    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:57:55.277023 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.277000    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:57:55.351672 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.351642    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-data\") pod \"561ecf04-5f23-49de-8881-639fc816fa81\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") "
Nov 03 19:57:55.351855 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.351680    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-socket\") pod \"561ecf04-5f23-49de-8881-639fc816fa81\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") "
Nov 03 19:57:55.351855 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.351732    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-envoy\") pod \"561ecf04-5f23-49de-8881-639fc816fa81\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") "
Nov 03 19:57:55.351855 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.351786    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-certs\") pod \"561ecf04-5f23-49de-8881-639fc816fa81\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") "
Nov 03 19:57:55.351855 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.351816    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/561ecf04-5f23-49de-8881-639fc816fa81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/561ecf04-5f23-49de-8881-639fc816fa81-istio-podinfo\") pod \"561ecf04-5f23-49de-8881-639fc816fa81\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") "
Nov 03 19:57:55.352071 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.351877    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-istio-token\" (UniqueName: \"kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-istio-token\") pod \"561ecf04-5f23-49de-8881-639fc816fa81\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") "
Nov 03 19:57:55.352071 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.351912    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561ecf04-5f23-49de-8881-639fc816fa81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/561ecf04-5f23-49de-8881-639fc816fa81-istiod-ca-cert\") pod \"561ecf04-5f23-49de-8881-639fc816fa81\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") "
Nov 03 19:57:55.352071 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.351987    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-credential-socket\") pod \"561ecf04-5f23-49de-8881-639fc816fa81\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") "
Nov 03 19:57:55.352071 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.352003    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-socket" pod "561ecf04-5f23-49de-8881-639fc816fa81" (UID: "561ecf04-5f23-49de-8881-639fc816fa81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:55.352262 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.352221    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-data" pod "561ecf04-5f23-49de-8881-639fc816fa81" (UID: "561ecf04-5f23-49de-8881-639fc816fa81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:55.352262 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.352234    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-credential-socket" pod "561ecf04-5f23-49de-8881-639fc816fa81" (UID: "561ecf04-5f23-49de-8881-639fc816fa81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:55.352262 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.352250    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-certs" pod "561ecf04-5f23-49de-8881-639fc816fa81" (UID: "561ecf04-5f23-49de-8881-639fc816fa81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:55.352427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.352263    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-kube-api-access-8pvst\" (UniqueName: \"kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-kube-api-access-8pvst\") pod \"561ecf04-5f23-49de-8881-639fc816fa81\" (UID: \"561ecf04-5f23-49de-8881-639fc816fa81\") "
Nov 03 19:57:55.352585 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.352548    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561ecf04-5f23-49de-8881-639fc816fa81-istiod-ca-cert" pod "561ecf04-5f23-49de-8881-639fc816fa81" (UID: "561ecf04-5f23-49de-8881-639fc816fa81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:55.352898 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.352879    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:55.352993 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.352902    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/561ecf04-5f23-49de-8881-639fc816fa81-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:55.352993 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.352917    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:55.352993 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.352932    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:55.352993 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.352946    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:55.354564 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.354539    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-envoy" pod "561ecf04-5f23-49de-8881-639fc816fa81" (UID: "561ecf04-5f23-49de-8881-639fc816fa81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:55.369842 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.369816    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/561ecf04-5f23-49de-8881-639fc816fa81-istio-podinfo" pod "561ecf04-5f23-49de-8881-639fc816fa81" (UID: "561ecf04-5f23-49de-8881-639fc816fa81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:57:55.369928 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.369860    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-istio-token" pod "561ecf04-5f23-49de-8881-639fc816fa81" (UID: "561ecf04-5f23-49de-8881-639fc816fa81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:55.371255 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.371225    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-kube-api-access-8pvst" pod "561ecf04-5f23-49de-8881-639fc816fa81" (UID: "561ecf04-5f23-49de-8881-639fc816fa81"). InnerVolumeSpecName "kube-api-access-8pvst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:55.455902 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.455858    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pvst\" (UniqueName: \"kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-kube-api-access-8pvst\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:55.455902 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.455901    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/561ecf04-5f23-49de-8881-639fc816fa81-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:55.455902 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.455916    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/561ecf04-5f23-49de-8881-639fc816fa81-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:55.456412 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.455934    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/561ecf04-5f23-49de-8881-639fc816fa81-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:55.501953 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.501873    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:55.719451 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:55.719415    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:56.041038 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.041005    5659 generic.go:356] "Generic (PLEG): container finished" podID="561ecf04-5f23-49de-8881-639fc816fa81" containerID="b27670d4c7a9eb0b406f95b19cd9cf735a435ae5733950ac2713f8bff8d1aa63" exitCode=0
Nov 03 19:57:56.041439 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.041083    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"
Nov 03 19:57:56.041439 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.041085    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" event={"ID":"561ecf04-5f23-49de-8881-639fc816fa81","Type":"ContainerDied","Data":"b27670d4c7a9eb0b406f95b19cd9cf735a435ae5733950ac2713f8bff8d1aa63"}
Nov 03 19:57:56.041439 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.041176    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr" event={"ID":"561ecf04-5f23-49de-8881-639fc816fa81","Type":"ContainerDied","Data":"17a176f9c75dfb0d881f9394d08d23477ebc8d99955395ab7e0765519751d84d"}
Nov 03 19:57:56.041439 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.041204    5659 scope.go:122] "RemoveContainer" containerID="b27670d4c7a9eb0b406f95b19cd9cf735a435ae5733950ac2713f8bff8d1aa63"
Nov 03 19:57:56.051711 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.051693    5659 scope.go:122] "RemoveContainer" containerID="a6d4f53c0f90986fe14e3c56f54098237bbb171256da82b36b7f36ca7323de5e"
Nov 03 19:57:56.061524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.061500    5659 scope.go:122] "RemoveContainer" containerID="b27670d4c7a9eb0b406f95b19cd9cf735a435ae5733950ac2713f8bff8d1aa63"
Nov 03 19:57:56.061836 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:57:56.061808    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27670d4c7a9eb0b406f95b19cd9cf735a435ae5733950ac2713f8bff8d1aa63\": container with ID starting with b27670d4c7a9eb0b406f95b19cd9cf735a435ae5733950ac2713f8bff8d1aa63 not found: ID does not exist" containerID="b27670d4c7a9eb0b406f95b19cd9cf735a435ae5733950ac2713f8bff8d1aa63"
Nov 03 19:57:56.061939 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.061837    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27670d4c7a9eb0b406f95b19cd9cf735a435ae5733950ac2713f8bff8d1aa63"} err="failed to get container status \"b27670d4c7a9eb0b406f95b19cd9cf735a435ae5733950ac2713f8bff8d1aa63\": rpc error: code = NotFound desc = could not find container \"b27670d4c7a9eb0b406f95b19cd9cf735a435ae5733950ac2713f8bff8d1aa63\": container with ID starting with b27670d4c7a9eb0b406f95b19cd9cf735a435ae5733950ac2713f8bff8d1aa63 not found: ID does not exist"
Nov 03 19:57:56.061939 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.061855    5659 scope.go:122] "RemoveContainer" containerID="a6d4f53c0f90986fe14e3c56f54098237bbb171256da82b36b7f36ca7323de5e"
Nov 03 19:57:56.062194 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:57:56.062161    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d4f53c0f90986fe14e3c56f54098237bbb171256da82b36b7f36ca7323de5e\": container with ID starting with a6d4f53c0f90986fe14e3c56f54098237bbb171256da82b36b7f36ca7323de5e not found: ID does not exist" containerID="a6d4f53c0f90986fe14e3c56f54098237bbb171256da82b36b7f36ca7323de5e"
Nov 03 19:57:56.062253 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.062189    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d4f53c0f90986fe14e3c56f54098237bbb171256da82b36b7f36ca7323de5e"} err="failed to get container status \"a6d4f53c0f90986fe14e3c56f54098237bbb171256da82b36b7f36ca7323de5e\": rpc error: code = NotFound desc = could not find container \"a6d4f53c0f90986fe14e3c56f54098237bbb171256da82b36b7f36ca7323de5e\": container with ID starting with a6d4f53c0f90986fe14e3c56f54098237bbb171256da82b36b7f36ca7323de5e not found: ID does not exist"
Nov 03 19:57:56.067014 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.066993    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"]
Nov 03 19:57:56.074891 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.074866    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9a6d4b-openshift-default-6ff4845f87-fqvmr"]
Nov 03 19:57:56.402551 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.402445    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"]
Nov 03 19:57:56.403454 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.403427    5659 cpu_manager.go:414] "containerMap: removing container" podUID="561ecf04-5f23-49de-8881-639fc816fa81" containerName="istio-proxy"
Nov 03 19:57:56.403694 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.403676    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561ecf04-5f23-49de-8881-639fc816fa81" containerName="istio-proxy"
Nov 03 19:57:56.403799 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.403789    5659 cpu_manager.go:414] "containerMap: removing container" podUID="561ecf04-5f23-49de-8881-639fc816fa81" containerName="istio-proxy"
Nov 03 19:57:56.403884 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.403875    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561ecf04-5f23-49de-8881-639fc816fa81" containerName="istio-proxy"
Nov 03 19:57:56.404095 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.404084    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="561ecf04-5f23-49de-8881-639fc816fa81" containerName="istio-proxy"
Nov 03 19:57:56.404181 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.404172    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="561ecf04-5f23-49de-8881-639fc816fa81" containerName="istio-proxy"
Nov 03 19:57:56.409737 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.409714    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:56.418335 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.416984    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-29f633-serving-cert\""
Nov 03 19:57:56.418335 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.417133    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"]
Nov 03 19:57:56.471445 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.471396    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd2tz\" (UniqueName: \"kubernetes.io/projected/53360c4d-e89f-45e5-9b77-4df72a1f93d0-kube-api-access-pd2tz\") pod \"maas-api-e2e-api-b-29f633-67f86bfb74-mds9g\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:56.471716 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.471687    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-maas-api-tls\") pod \"maas-api-e2e-api-b-29f633-67f86bfb74-mds9g\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:56.471871 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.471822    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53360c4d-e89f-45e5-9b77-4df72a1f93d0-openshift-service-ca\") pod \"maas-api-e2e-api-b-29f633-67f86bfb74-mds9g\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:56.471920 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.471905    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-metrics-certs\") pod \"maas-api-e2e-api-b-29f633-67f86bfb74-mds9g\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:56.530439 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.530408    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:56.573243 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.573205    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd2tz\" (UniqueName: \"kubernetes.io/projected/53360c4d-e89f-45e5-9b77-4df72a1f93d0-kube-api-access-pd2tz\") pod \"maas-api-e2e-api-b-29f633-67f86bfb74-mds9g\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:56.573458 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.573274    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-maas-api-tls\") pod \"maas-api-e2e-api-b-29f633-67f86bfb74-mds9g\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:56.573458 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.573402    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53360c4d-e89f-45e5-9b77-4df72a1f93d0-openshift-service-ca\") pod \"maas-api-e2e-api-b-29f633-67f86bfb74-mds9g\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:56.573635 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.573476    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-metrics-certs\") pod \"maas-api-e2e-api-b-29f633-67f86bfb74-mds9g\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:56.574063 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.574035    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53360c4d-e89f-45e5-9b77-4df72a1f93d0-openshift-service-ca\") pod \"maas-api-e2e-api-b-29f633-67f86bfb74-mds9g\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:56.575873 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.575851    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-maas-api-tls\") pod \"maas-api-e2e-api-b-29f633-67f86bfb74-mds9g\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:56.576019 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.575993    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-metrics-certs\") pod \"maas-api-e2e-api-b-29f633-67f86bfb74-mds9g\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:56.591272 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.591242    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd2tz\" (UniqueName: \"kubernetes.io/projected/53360c4d-e89f-45e5-9b77-4df72a1f93d0-kube-api-access-pd2tz\") pod \"maas-api-e2e-api-b-29f633-67f86bfb74-mds9g\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:56.730528 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.722919    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:56.744507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:56.740088    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:57.180878 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:57.180837    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"]
Nov 03 19:57:57.181964 ip-10-0-133-249 kubenswrapper[5659]: W1103 19:57:57.181932    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53360c4d_e89f_45e5_9b77_4df72a1f93d0.slice/crio-17b88166c4a7fa4ac0161305555730dc126ac516573835654763289aec6f0bce WatchSource:0}: Error finding container 17b88166c4a7fa4ac0161305555730dc126ac516573835654763289aec6f0bce: Status 404 returned error can't find the container with id 17b88166c4a7fa4ac0161305555730dc126ac516573835654763289aec6f0bce
Nov 03 19:57:57.934042 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:57.934009    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:57.971513 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:57.971447    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="561ecf04-5f23-49de-8881-639fc816fa81" path="/var/lib/kubelet/pods/561ecf04-5f23-49de-8881-639fc816fa81/volumes"
Nov 03 19:57:57.984768 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:57.984742    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"
Nov 03 19:57:57.985091 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:57.985075    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"
Nov 03 19:57:58.052814 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.052726    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"]
Nov 03 19:57:58.058210 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.058181    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"
Nov 03 19:57:58.067338 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.064418    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-1b1d04\""
Nov 03 19:57:58.067338 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.064721    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g" event={"ID":"53360c4d-e89f-45e5-9b77-4df72a1f93d0","Type":"ContainerStarted","Data":"07db43dd8651de1032ed41e7a7ea0fd2fde3085a99a813e453c0956d396056af"}
Nov 03 19:57:58.067338 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.064775    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g" event={"ID":"53360c4d-e89f-45e5-9b77-4df72a1f93d0","Type":"ContainerStarted","Data":"17b88166c4a7fa4ac0161305555730dc126ac516573835654763289aec6f0bce"}
Nov 03 19:57:58.067338 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.066742    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:57:58.083353 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.083325    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"]
Nov 03 19:57:58.104827 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.104794    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcc86d4-6bdb-433d-9c41-f14f00985280-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk\" (UID: \"abcc86d4-6bdb-433d-9c41-f14f00985280\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"
Nov 03 19:57:58.105197 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.105167    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk9r6\" (UniqueName: \"kubernetes.io/projected/abcc86d4-6bdb-433d-9c41-f14f00985280-kube-api-access-sk9r6\") pod \"payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk\" (UID: \"abcc86d4-6bdb-433d-9c41-f14f00985280\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"
Nov 03 19:57:58.108306 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.108233    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g" podStartSLOduration=1.531436313 podStartE2EDuration="2.10821703s" totalImagesPullingTime="576.780717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:58.101911255 +0000 UTC m=+1230.256813675" watchObservedRunningTime="2026-11-03 19:57:58.10821703 +0000 UTC m=+1230.263119450"
Nov 03 19:57:58.137604 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.137569    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:58.160262 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.160226    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-1b1d04-dockercfg-kjs5n\""
Nov 03 19:57:58.206321 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.206279    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9r6\" (UniqueName: \"kubernetes.io/projected/abcc86d4-6bdb-433d-9c41-f14f00985280-kube-api-access-sk9r6\") pod \"payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk\" (UID: \"abcc86d4-6bdb-433d-9c41-f14f00985280\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"
Nov 03 19:57:58.206803 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.206389    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcc86d4-6bdb-433d-9c41-f14f00985280-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk\" (UID: \"abcc86d4-6bdb-433d-9c41-f14f00985280\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"
Nov 03 19:57:58.207111 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.207092    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcc86d4-6bdb-433d-9c41-f14f00985280-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk\" (UID: \"abcc86d4-6bdb-433d-9c41-f14f00985280\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"
Nov 03 19:57:58.226507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.226433    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9r6\" (UniqueName: \"kubernetes.io/projected/abcc86d4-6bdb-433d-9c41-f14f00985280-kube-api-access-sk9r6\") pod \"payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk\" (UID: \"abcc86d4-6bdb-433d-9c41-f14f00985280\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"
Nov 03 19:57:58.307562 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.307419    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:58.403911 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.403875    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"
Nov 03 19:57:58.772114 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:58.772081    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"]
Nov 03 19:57:59.072985 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:59.072893    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk" event={"ID":"abcc86d4-6bdb-433d-9c41-f14f00985280","Type":"ContainerStarted","Data":"dd8485b4faf08de9f501e353b9aebc1aa76e5ef0ce084ef9be53485fb26c9280"}
Nov 03 19:57:59.072985 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:59.072965    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"
Nov 03 19:57:59.073206 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:59.072990    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk" event={"ID":"abcc86d4-6bdb-433d-9c41-f14f00985280","Type":"ContainerStarted","Data":"bd01b4bef9053d732ff7a7c6eedcef21343d44a1dd45ddf15a582db2f345332e"}
Nov 03 19:57:59.099345 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:59.099284    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk" podStartSLOduration=1.099272063 podStartE2EDuration="1.099272063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:59.09524564 +0000 UTC m=+1231.250148058" watchObservedRunningTime="2026-11-03 19:57:59.099272063 +0000 UTC m=+1231.254174482"
Nov 03 19:57:59.366284 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:57:59.366026    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:00.570631 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:00.570457    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:01.383382 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:01.383339    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:02.284506 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:02.284016    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:02.595207 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:02.595093    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:02.897036 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:02.896879    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:03.027612 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:03.027582    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:03.170415 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:03.170322    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:03.649994 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:03.649961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:04.419800 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:04.419772    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:05.087811 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:05.087781    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:58:05.260629 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:05.260583    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:06.155564 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:06.142641    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:06.996228 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:06.994644    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:08.072159 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:08.072126    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:09.314933 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:09.314889    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:09.452351 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:09.452313    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:10.085224 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:10.085185    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"
Nov 03 19:58:10.839877 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:10.839844    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:12.010252 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:12.010216    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:13.546171 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:13.546136    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:14.186545 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:14.186511    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:14.792722 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:14.792688    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:15.217553 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:58:15.217518    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:58:15.365911 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:15.365879    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:15.704074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:15.704010    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:15.842062 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:15.842030    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:16.402942 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:16.402898    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:17.042248 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:17.042188    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:17.199399 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:17.199361    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:17.750013 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:17.749973    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:18.597314 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:18.597281    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:18.932032 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:18.931945    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:19.155116 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:19.155082    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:19.511427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:19.511392    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:19.657104 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:19.657070    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:20.037354 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:20.037314    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:20.841444 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:20.841411    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:21.035901 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:21.035871    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:21.878017 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:21.877975    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:21.942466 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:21.942431    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"]
Nov 03 19:58:22.136940 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:22.136853    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:22.305276 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:22.305237    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:23.432427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:23.432397    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:23.591817 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:23.591764    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:24.567320 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:24.567275    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:25.695842 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:25.695814    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:26.942848 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:26.942798    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9" podUID="670fc60a-c585-4c83-a527-143c4c951f89" containerName="payload-processing" containerID="cri-o://1fa9faf33bc5c45ca29e7e3ca950e4c3802dffe794a1ab87e62b343c4a71a0fe" gracePeriod=25
Nov 03 19:58:27.058327 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.058267    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:27.247820 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.247795    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"
Nov 03 19:58:27.297326 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.297290    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670fc60a-c585-4c83-a527-143c4c951f89-kube-api-access-8hzg4\" (UniqueName: \"kubernetes.io/projected/670fc60a-c585-4c83-a527-143c4c951f89-kube-api-access-8hzg4\") pod \"670fc60a-c585-4c83-a527-143c4c951f89\" (UID: \"670fc60a-c585-4c83-a527-143c4c951f89\") "
Nov 03 19:58:27.297519 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.297447    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/670fc60a-c585-4c83-a527-143c4c951f89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670fc60a-c585-4c83-a527-143c4c951f89-plugins-config-volume\") pod \"670fc60a-c585-4c83-a527-143c4c951f89\" (UID: \"670fc60a-c585-4c83-a527-143c4c951f89\") "
Nov 03 19:58:27.297849 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.297822    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670fc60a-c585-4c83-a527-143c4c951f89-plugins-config-volume" pod "670fc60a-c585-4c83-a527-143c4c951f89" (UID: "670fc60a-c585-4c83-a527-143c4c951f89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:27.298127 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.298104    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670fc60a-c585-4c83-a527-143c4c951f89-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:27.300227 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.300197    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670fc60a-c585-4c83-a527-143c4c951f89-kube-api-access-8hzg4" pod "670fc60a-c585-4c83-a527-143c4c951f89" (UID: "670fc60a-c585-4c83-a527-143c4c951f89"). InnerVolumeSpecName "kube-api-access-8hzg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:27.372741 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.372712    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:27.382230 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.382200    5659 generic.go:356] "Generic (PLEG): container finished" podID="670fc60a-c585-4c83-a527-143c4c951f89" containerID="1fa9faf33bc5c45ca29e7e3ca950e4c3802dffe794a1ab87e62b343c4a71a0fe" exitCode=0
Nov 03 19:58:27.382504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.382262    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"
Nov 03 19:58:27.382504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.382286    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9" event={"ID":"670fc60a-c585-4c83-a527-143c4c951f89","Type":"ContainerDied","Data":"1fa9faf33bc5c45ca29e7e3ca950e4c3802dffe794a1ab87e62b343c4a71a0fe"}
Nov 03 19:58:27.382504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.382347    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9" event={"ID":"670fc60a-c585-4c83-a527-143c4c951f89","Type":"ContainerDied","Data":"622fb61c729a79d6d03c87848eea52b7b27d35704ac8a22a5977b0f5f189aac8"}
Nov 03 19:58:27.382504 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.382373    5659 scope.go:122] "RemoveContainer" containerID="1fa9faf33bc5c45ca29e7e3ca950e4c3802dffe794a1ab87e62b343c4a71a0fe"
Nov 03 19:58:27.393041 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.393024    5659 scope.go:122] "RemoveContainer" containerID="1fa9faf33bc5c45ca29e7e3ca950e4c3802dffe794a1ab87e62b343c4a71a0fe"
Nov 03 19:58:27.393326 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:58:27.393304    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa9faf33bc5c45ca29e7e3ca950e4c3802dffe794a1ab87e62b343c4a71a0fe\": container with ID starting with 1fa9faf33bc5c45ca29e7e3ca950e4c3802dffe794a1ab87e62b343c4a71a0fe not found: ID does not exist" containerID="1fa9faf33bc5c45ca29e7e3ca950e4c3802dffe794a1ab87e62b343c4a71a0fe"
Nov 03 19:58:27.393383 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.393335    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa9faf33bc5c45ca29e7e3ca950e4c3802dffe794a1ab87e62b343c4a71a0fe"} err="failed to get container status \"1fa9faf33bc5c45ca29e7e3ca950e4c3802dffe794a1ab87e62b343c4a71a0fe\": rpc error: code = NotFound desc = could not find container \"1fa9faf33bc5c45ca29e7e3ca950e4c3802dffe794a1ab87e62b343c4a71a0fe\": container with ID starting with 1fa9faf33bc5c45ca29e7e3ca950e4c3802dffe794a1ab87e62b343c4a71a0fe not found: ID does not exist"
Nov 03 19:58:27.399718 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.399697    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hzg4\" (UniqueName: \"kubernetes.io/projected/670fc60a-c585-4c83-a527-143c4c951f89-kube-api-access-8hzg4\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:27.406166 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.406143    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"]
Nov 03 19:58:27.410390 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.410372    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a7dc9-5cc9bf64f4-ttrz9"]
Nov 03 19:58:27.973089 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:27.973052    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="670fc60a-c585-4c83-a527-143c4c951f89" path="/var/lib/kubelet/pods/670fc60a-c585-4c83-a527-143c4c951f89/volumes"
Nov 03 19:58:28.290793 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:28.290708    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:29.255001 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:29.254974    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:29.502839 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:29.502807    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:30.219673 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:30.219640    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:31.483830 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:31.483762    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:32.132256 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:32.132227    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:32.728131 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:32.728095    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:33.138463 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:33.138378    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:33.286055 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:33.286025    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:33.453783 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:33.453701    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:33.709118 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:33.709037    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:33.871507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:33.863688    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:34.416842 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:34.416806    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:35.102505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:35.102457    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:35.242016 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:35.241968    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:35.572226 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:35.572158    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:36.011199 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:36.010953    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:36.218834 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:36.218740    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:36.362291 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:36.362191    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:37.224894 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:37.224828    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:37.369007 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:37.368966    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:37.869613 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:37.869582    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:38.108305 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:38.108263    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:38.256068 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:38.256020    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:38.395844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:38.395769    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:38.807375 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:38.807164    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:38.955775 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:38.955654    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:39.086968 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:39.086924    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:39.562475 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:39.562438    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:39.824457 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:39.824371    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:40.049940 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:40.049888    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:40.845937 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:40.845902    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:41.008608 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:41.008576    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:41.098068 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:41.097622    5659 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:58:41.098068 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:41.097702    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" podUID="74da69e9-a168-4ae5-a882-daca94333466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:41.150401 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:41.150292    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:42.272191 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:42.272156    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:43.276922 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:43.276869    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:43.573581 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:43.573462    5659 generic.go:356] "Generic (PLEG): container finished" podID="74da69e9-a168-4ae5-a882-daca94333466" containerID="96ee96b4fb32413f0f4a13240ff155b1f3e23752fb2bf428b899ad41fb553693" exitCode=0
Nov 03 19:58:43.573581 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:43.573546    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" event={"ID":"74da69e9-a168-4ae5-a882-daca94333466","Type":"ContainerDied","Data":"96ee96b4fb32413f0f4a13240ff155b1f3e23752fb2bf428b899ad41fb553693"}
Nov 03 19:58:43.574133 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:43.574109    5659 scope.go:122] "RemoveContainer" containerID="96ee96b4fb32413f0f4a13240ff155b1f3e23752fb2bf428b899ad41fb553693"
Nov 03 19:58:43.578059 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:43.578011    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:43.578173 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:43.578110    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:43.578173 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:43.578156    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:43.940196 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:43.940091    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:44.088606 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:44.087474    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:44.253515 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:44.253379    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:44.410427 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:44.410400    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:44.582972 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:44.582883    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" event={"ID":"74da69e9-a168-4ae5-a882-daca94333466","Type":"ContainerStarted","Data":"855f4d3259192e4d9afefaaed5ed9b0941633808c49dc562d6e0ff01150f70de"}
Nov 03 19:58:44.782819 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:44.782785    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:44.969977 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:44.969939    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:58:44.970344 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:44.970322    5659 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:58:44.970521 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:44.970496    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" podUID="74da69e9-a168-4ae5-a882-daca94333466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 19:58:45.219764 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:58:45.219728    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:58:45.483022 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:45.482991    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:45.969896 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:45.969865    5659 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:58:45.970080 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:45.969916    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" podUID="74da69e9-a168-4ae5-a882-daca94333466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 19:58:46.422067 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:46.421986    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:46.616403 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:46.616367    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:46.973913 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:46.973877    5659 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:58:46.974199 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:46.974172    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" podUID="74da69e9-a168-4ae5-a882-daca94333466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 19:58:47.174642 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:47.174604    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:47.499257 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:47.499228    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:47.836569 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:47.836142    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"]
Nov 03 19:58:47.876534 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:47.875527    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"]
Nov 03 19:58:47.973060 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:47.973024    5659 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:58:47.973280 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:47.973259    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" podUID="74da69e9-a168-4ae5-a882-daca94333466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 19:58:48.025691 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:48.025662    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:48.179052 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:48.178956    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:48.319093 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:48.319054    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:48.977736 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:48.977694    5659 patch_prober.go:29] interesting pod/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:58:48.978186 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:48.977775    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" podUID="74da69e9-a168-4ae5-a882-daca94333466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 19:58:49.061071 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:49.061021    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:49.658932 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:49.658898    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:49.985853 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:49.985826    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:58:49.986282 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:49.986256    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:58:49.987140 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:49.987119    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 19:58:50.359028 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:50.358939    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"]
Nov 03 19:58:50.359202 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:50.359138    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2" podUID="0dde3468-dceb-4d4e-8e71-1eba7e214145" containerName="istio-proxy" containerID="cri-o://ee578c20e6a5cfbaebcede3bd07d17398fb12d4d1999375500d3b9ef87e29c1e" gracePeriod=30
Nov 03 19:58:50.362186 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:50.362158    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"]
Nov 03 19:58:50.362358 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:50.362339    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" containerName="istio-proxy" containerID="cri-o://21abac9cea58740d3e74430a303aae277c5870c67bf42a66de51adc0dfb8ee2a" gracePeriod=30
Nov 03 19:58:50.956973 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:50.956935    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:51.277977 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:51.277895    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:51.595218 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:51.595137    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:52.546848 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:52.546818    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:52.745985 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:52.745956    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:52.837556 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:52.837439    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk" podUID="30199a75-1c40-43e5-9c62-a279c19890d4" containerName="payload-processing" containerID="cri-o://59be52ab2562757d6aca43c60814e6e1f658bbf83e6173105aaa6ae924e470b3" gracePeriod=25
Nov 03 19:58:52.876148 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:52.876106    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg" podUID="6dbac47c-533a-4e63-b65b-c87a2a6eee14" containerName="payload-pre-processing" containerID="cri-o://5fe544f3c9310abaed43ec298bcaa449caa49e4761ea5ac8fef88daee316243c" gracePeriod=25
Nov 03 19:58:53.002704 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.002675    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:53.423301 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.423223    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"
Nov 03 19:58:53.503040 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.503009    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30199a75-1c40-43e5-9c62-a279c19890d4-kube-api-access-q5dnc\" (UniqueName: \"kubernetes.io/projected/30199a75-1c40-43e5-9c62-a279c19890d4-kube-api-access-q5dnc\") pod \"30199a75-1c40-43e5-9c62-a279c19890d4\" (UID: \"30199a75-1c40-43e5-9c62-a279c19890d4\") "
Nov 03 19:58:53.503870 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.503846    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30199a75-1c40-43e5-9c62-a279c19890d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30199a75-1c40-43e5-9c62-a279c19890d4-plugins-config-volume\") pod \"30199a75-1c40-43e5-9c62-a279c19890d4\" (UID: \"30199a75-1c40-43e5-9c62-a279c19890d4\") "
Nov 03 19:58:53.504447 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.504332    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30199a75-1c40-43e5-9c62-a279c19890d4-plugins-config-volume" pod "30199a75-1c40-43e5-9c62-a279c19890d4" (UID: "30199a75-1c40-43e5-9c62-a279c19890d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:53.505194 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.505170    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30199a75-1c40-43e5-9c62-a279c19890d4-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:53.513131 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.513093    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30199a75-1c40-43e5-9c62-a279c19890d4-kube-api-access-q5dnc" pod "30199a75-1c40-43e5-9c62-a279c19890d4" (UID: "30199a75-1c40-43e5-9c62-a279c19890d4"). InnerVolumeSpecName "kube-api-access-q5dnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:53.514866 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.514562    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"
Nov 03 19:58:53.617511 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.613712    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5dnc\" (UniqueName: \"kubernetes.io/projected/30199a75-1c40-43e5-9c62-a279c19890d4-kube-api-access-q5dnc\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:53.690972 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.690935    5659 generic.go:356] "Generic (PLEG): container finished" podID="30199a75-1c40-43e5-9c62-a279c19890d4" containerID="59be52ab2562757d6aca43c60814e6e1f658bbf83e6173105aaa6ae924e470b3" exitCode=0
Nov 03 19:58:53.691232 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.690982    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"
Nov 03 19:58:53.691428 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.690999    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk" event={"ID":"30199a75-1c40-43e5-9c62-a279c19890d4","Type":"ContainerDied","Data":"59be52ab2562757d6aca43c60814e6e1f658bbf83e6173105aaa6ae924e470b3"}
Nov 03 19:58:53.691622 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.691469    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk" event={"ID":"30199a75-1c40-43e5-9c62-a279c19890d4","Type":"ContainerDied","Data":"d16d3307fc0fcfcfc857127cd6b2ef02ee17417f0110fcb458d82fb90ebfc63c"}
Nov 03 19:58:53.691622 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.691543    5659 scope.go:122] "RemoveContainer" containerID="59be52ab2562757d6aca43c60814e6e1f658bbf83e6173105aaa6ae924e470b3"
Nov 03 19:58:53.695808 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.695469    5659 generic.go:356] "Generic (PLEG): container finished" podID="6dbac47c-533a-4e63-b65b-c87a2a6eee14" containerID="5fe544f3c9310abaed43ec298bcaa449caa49e4761ea5ac8fef88daee316243c" exitCode=0
Nov 03 19:58:53.695808 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.695541    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg" event={"ID":"6dbac47c-533a-4e63-b65b-c87a2a6eee14","Type":"ContainerDied","Data":"5fe544f3c9310abaed43ec298bcaa449caa49e4761ea5ac8fef88daee316243c"}
Nov 03 19:58:53.695808 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.695590    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg" event={"ID":"6dbac47c-533a-4e63-b65b-c87a2a6eee14","Type":"ContainerDied","Data":"c45a8f464670c83a3497d1dc1ffcc4cf0573c9f34265009af65717851cd99e10"}
Nov 03 19:58:53.695808 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.695662    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"
Nov 03 19:58:53.716523 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.714935    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbac47c-533a-4e63-b65b-c87a2a6eee14-kube-api-access-fltj5\" (UniqueName: \"kubernetes.io/projected/6dbac47c-533a-4e63-b65b-c87a2a6eee14-kube-api-access-fltj5\") pod \"6dbac47c-533a-4e63-b65b-c87a2a6eee14\" (UID: \"6dbac47c-533a-4e63-b65b-c87a2a6eee14\") "
Nov 03 19:58:53.716523 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.715070    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dbac47c-533a-4e63-b65b-c87a2a6eee14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbac47c-533a-4e63-b65b-c87a2a6eee14-plugins-config-volume\") pod \"6dbac47c-533a-4e63-b65b-c87a2a6eee14\" (UID: \"6dbac47c-533a-4e63-b65b-c87a2a6eee14\") "
Nov 03 19:58:53.717568 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.717165    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbac47c-533a-4e63-b65b-c87a2a6eee14-plugins-config-volume" pod "6dbac47c-533a-4e63-b65b-c87a2a6eee14" (UID: "6dbac47c-533a-4e63-b65b-c87a2a6eee14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:53.753554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.727885    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbac47c-533a-4e63-b65b-c87a2a6eee14-kube-api-access-fltj5" pod "6dbac47c-533a-4e63-b65b-c87a2a6eee14" (UID: "6dbac47c-533a-4e63-b65b-c87a2a6eee14"). InnerVolumeSpecName "kube-api-access-fltj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:53.753554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.747740    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"]
Nov 03 19:58:53.753554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.750300    5659 scope.go:122] "RemoveContainer" containerID="59be52ab2562757d6aca43c60814e6e1f658bbf83e6173105aaa6ae924e470b3"
Nov 03 19:58:53.753554 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:58:53.751153    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59be52ab2562757d6aca43c60814e6e1f658bbf83e6173105aaa6ae924e470b3\": container with ID starting with 59be52ab2562757d6aca43c60814e6e1f658bbf83e6173105aaa6ae924e470b3 not found: ID does not exist" containerID="59be52ab2562757d6aca43c60814e6e1f658bbf83e6173105aaa6ae924e470b3"
Nov 03 19:58:53.753554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.751197    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59be52ab2562757d6aca43c60814e6e1f658bbf83e6173105aaa6ae924e470b3"} err="failed to get container status \"59be52ab2562757d6aca43c60814e6e1f658bbf83e6173105aaa6ae924e470b3\": rpc error: code = NotFound desc = could not find container \"59be52ab2562757d6aca43c60814e6e1f658bbf83e6173105aaa6ae924e470b3\": container with ID starting with 59be52ab2562757d6aca43c60814e6e1f658bbf83e6173105aaa6ae924e470b3 not found: ID does not exist"
Nov 03 19:58:53.753554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.751226    5659 scope.go:122] "RemoveContainer" containerID="5fe544f3c9310abaed43ec298bcaa449caa49e4761ea5ac8fef88daee316243c"
Nov 03 19:58:53.767679 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.767646    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-29f633-fb468b56b-ns5qk"]
Nov 03 19:58:53.768604 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.768583    5659 scope.go:122] "RemoveContainer" containerID="5fe544f3c9310abaed43ec298bcaa449caa49e4761ea5ac8fef88daee316243c"
Nov 03 19:58:53.768961 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:58:53.768936    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe544f3c9310abaed43ec298bcaa449caa49e4761ea5ac8fef88daee316243c\": container with ID starting with 5fe544f3c9310abaed43ec298bcaa449caa49e4761ea5ac8fef88daee316243c not found: ID does not exist" containerID="5fe544f3c9310abaed43ec298bcaa449caa49e4761ea5ac8fef88daee316243c"
Nov 03 19:58:53.769037 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.768975    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe544f3c9310abaed43ec298bcaa449caa49e4761ea5ac8fef88daee316243c"} err="failed to get container status \"5fe544f3c9310abaed43ec298bcaa449caa49e4761ea5ac8fef88daee316243c\": rpc error: code = NotFound desc = could not find container \"5fe544f3c9310abaed43ec298bcaa449caa49e4761ea5ac8fef88daee316243c\": container with ID starting with 5fe544f3c9310abaed43ec298bcaa449caa49e4761ea5ac8fef88daee316243c not found: ID does not exist"
Nov 03 19:58:53.817810 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.817737    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fltj5\" (UniqueName: \"kubernetes.io/projected/6dbac47c-533a-4e63-b65b-c87a2a6eee14-kube-api-access-fltj5\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:53.817810 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.817768    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbac47c-533a-4e63-b65b-c87a2a6eee14-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:53.971377 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:53.971339    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30199a75-1c40-43e5-9c62-a279c19890d4" path="/var/lib/kubelet/pods/30199a75-1c40-43e5-9c62-a279c19890d4/volumes"
Nov 03 19:58:54.034790 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:54.034717    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"]
Nov 03 19:58:54.041772 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:54.041742    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-29f633-6f8fb58955-x6jlg"]
Nov 03 19:58:54.374894 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:54.360886    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:55.554360 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.553599    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:55.702685 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.702361    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:55.723122 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.721981    5659 generic.go:356] "Generic (PLEG): container finished" podID="1d34e5e5-88b0-4040-9353-840bd249d790" containerID="21abac9cea58740d3e74430a303aae277c5870c67bf42a66de51adc0dfb8ee2a" exitCode=0
Nov 03 19:58:55.723122 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.722186    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" event={"ID":"1d34e5e5-88b0-4040-9353-840bd249d790","Type":"ContainerDied","Data":"21abac9cea58740d3e74430a303aae277c5870c67bf42a66de51adc0dfb8ee2a"}
Nov 03 19:58:55.724031 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.724011    5659 generic.go:356] "Generic (PLEG): container finished" podID="0dde3468-dceb-4d4e-8e71-1eba7e214145" containerID="ee578c20e6a5cfbaebcede3bd07d17398fb12d4d1999375500d3b9ef87e29c1e" exitCode=0
Nov 03 19:58:55.724151 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.724066    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2" event={"ID":"0dde3468-dceb-4d4e-8e71-1eba7e214145","Type":"ContainerDied","Data":"ee578c20e6a5cfbaebcede3bd07d17398fb12d4d1999375500d3b9ef87e29c1e"}
Nov 03 19:58:55.724151 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.724096    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2" event={"ID":"0dde3468-dceb-4d4e-8e71-1eba7e214145","Type":"ContainerDied","Data":"923a036e62ba3c07dca700db556fa301cfdff1c2246540589bba5297fd68b44e"}
Nov 03 19:58:55.724151 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.724111    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="923a036e62ba3c07dca700db556fa301cfdff1c2246540589bba5297fd68b44e"
Nov 03 19:58:55.732092 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.732064    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:58:55.771537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.771446    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dde3468-dceb-4d4e-8e71-1eba7e214145-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dde3468-dceb-4d4e-8e71-1eba7e214145-istiod-ca-cert\") pod \"0dde3468-dceb-4d4e-8e71-1eba7e214145\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") "
Nov 03 19:58:55.771932 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.771901    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dde3468-dceb-4d4e-8e71-1eba7e214145-istiod-ca-cert" pod "0dde3468-dceb-4d4e-8e71-1eba7e214145" (UID: "0dde3468-dceb-4d4e-8e71-1eba7e214145"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:55.772537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.772391    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-socket\") pod \"0dde3468-dceb-4d4e-8e71-1eba7e214145\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") "
Nov 03 19:58:55.772537 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.772433    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-kube-api-access-xngl8\" (UniqueName: \"kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-kube-api-access-xngl8\") pod \"0dde3468-dceb-4d4e-8e71-1eba7e214145\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") "
Nov 03 19:58:55.774057 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.772737    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-token\" (UniqueName: \"kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-token\") pod \"0dde3468-dceb-4d4e-8e71-1eba7e214145\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") "
Nov 03 19:58:55.774057 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.772796    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-envoy\") pod \"0dde3468-dceb-4d4e-8e71-1eba7e214145\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") "
Nov 03 19:58:55.774057 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.772843    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-certs\") pod \"0dde3468-dceb-4d4e-8e71-1eba7e214145\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") "
Nov 03 19:58:55.774057 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.772977    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-podinfo\") pod \"0dde3468-dceb-4d4e-8e71-1eba7e214145\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") "
Nov 03 19:58:55.774057 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.773125    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-credential-socket\") pod \"0dde3468-dceb-4d4e-8e71-1eba7e214145\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") "
Nov 03 19:58:55.774057 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.773162    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-data\") pod \"0dde3468-dceb-4d4e-8e71-1eba7e214145\" (UID: \"0dde3468-dceb-4d4e-8e71-1eba7e214145\") "
Nov 03 19:58:55.774057 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.773843    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dde3468-dceb-4d4e-8e71-1eba7e214145-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.774546 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.774526    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-data" pod "0dde3468-dceb-4d4e-8e71-1eba7e214145" (UID: "0dde3468-dceb-4d4e-8e71-1eba7e214145"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:55.774835 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.774815    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-socket" pod "0dde3468-dceb-4d4e-8e71-1eba7e214145" (UID: "0dde3468-dceb-4d4e-8e71-1eba7e214145"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:55.775418 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.775308    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-certs" pod "0dde3468-dceb-4d4e-8e71-1eba7e214145" (UID: "0dde3468-dceb-4d4e-8e71-1eba7e214145"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:55.778074 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.776579    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-credential-socket" pod "0dde3468-dceb-4d4e-8e71-1eba7e214145" (UID: "0dde3468-dceb-4d4e-8e71-1eba7e214145"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:55.779171 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.779137    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-token" pod "0dde3468-dceb-4d4e-8e71-1eba7e214145" (UID: "0dde3468-dceb-4d4e-8e71-1eba7e214145"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:55.780325 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.780283    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-envoy" pod "0dde3468-dceb-4d4e-8e71-1eba7e214145" (UID: "0dde3468-dceb-4d4e-8e71-1eba7e214145"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:55.780475 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.780447    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-kube-api-access-xngl8" pod "0dde3468-dceb-4d4e-8e71-1eba7e214145" (UID: "0dde3468-dceb-4d4e-8e71-1eba7e214145"). InnerVolumeSpecName "kube-api-access-xngl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:55.781916 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.781890    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-podinfo" pod "0dde3468-dceb-4d4e-8e71-1eba7e214145" (UID: "0dde3468-dceb-4d4e-8e71-1eba7e214145"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:58:55.787876 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.787855    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:58:55.875077 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.875042    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-kube-api-access-bddwl\" (UniqueName: \"kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-kube-api-access-bddwl\") pod \"1d34e5e5-88b0-4040-9353-840bd249d790\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") "
Nov 03 19:58:55.875266 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.875123    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-socket\") pod \"1d34e5e5-88b0-4040-9353-840bd249d790\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") "
Nov 03 19:58:55.875266 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.875188    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-certs\") pod \"1d34e5e5-88b0-4040-9353-840bd249d790\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") "
Nov 03 19:58:55.875266 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.875223    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d34e5e5-88b0-4040-9353-840bd249d790-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d34e5e5-88b0-4040-9353-840bd249d790-istiod-ca-cert\") pod \"1d34e5e5-88b0-4040-9353-840bd249d790\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") "
Nov 03 19:58:55.875445 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.875405    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-istio-token\" (UniqueName: \"kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-istio-token\") pod \"1d34e5e5-88b0-4040-9353-840bd249d790\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") "
Nov 03 19:58:55.875550 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.875446    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-socket" pod "1d34e5e5-88b0-4040-9353-840bd249d790" (UID: "1d34e5e5-88b0-4040-9353-840bd249d790"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:55.875550 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.875503    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-data\") pod \"1d34e5e5-88b0-4040-9353-840bd249d790\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") "
Nov 03 19:58:55.875670 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.875554    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-certs" pod "1d34e5e5-88b0-4040-9353-840bd249d790" (UID: "1d34e5e5-88b0-4040-9353-840bd249d790"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:55.875670 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.875567    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d34e5e5-88b0-4040-9353-840bd249d790-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d34e5e5-88b0-4040-9353-840bd249d790-istio-podinfo\") pod \"1d34e5e5-88b0-4040-9353-840bd249d790\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") "
Nov 03 19:58:55.875670 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.875609    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-credential-socket\") pod \"1d34e5e5-88b0-4040-9353-840bd249d790\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") "
Nov 03 19:58:55.875670 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.875644    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-envoy\") pod \"1d34e5e5-88b0-4040-9353-840bd249d790\" (UID: \"1d34e5e5-88b0-4040-9353-840bd249d790\") "
Nov 03 19:58:55.875861 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.875757    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-data" pod "1d34e5e5-88b0-4040-9353-840bd249d790" (UID: "1d34e5e5-88b0-4040-9353-840bd249d790"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:55.875973 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.875953    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-credential-socket" pod "1d34e5e5-88b0-4040-9353-840bd249d790" (UID: "1d34e5e5-88b0-4040-9353-840bd249d790"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876091    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d34e5e5-88b0-4040-9353-840bd249d790-istiod-ca-cert" pod "1d34e5e5-88b0-4040-9353-840bd249d790" (UID: "1d34e5e5-88b0-4040-9353-840bd249d790"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876323    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876346    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876363    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876378    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d34e5e5-88b0-4040-9353-840bd249d790-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876393    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876407    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876421    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876436    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xngl8\" (UniqueName: \"kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-kube-api-access-xngl8\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876451    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876466    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876497    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876512    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dde3468-dceb-4d4e-8e71-1eba7e214145-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.876552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.876526    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.877820 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.877799    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-kube-api-access-bddwl" pod "1d34e5e5-88b0-4040-9353-840bd249d790" (UID: "1d34e5e5-88b0-4040-9353-840bd249d790"). InnerVolumeSpecName "kube-api-access-bddwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:55.878048 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.878028    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-istio-token" pod "1d34e5e5-88b0-4040-9353-840bd249d790" (UID: "1d34e5e5-88b0-4040-9353-840bd249d790"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:55.878311 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.878282    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-envoy" pod "1d34e5e5-88b0-4040-9353-840bd249d790" (UID: "1d34e5e5-88b0-4040-9353-840bd249d790"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:55.879008 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.878988    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d34e5e5-88b0-4040-9353-840bd249d790-istio-podinfo" pod "1d34e5e5-88b0-4040-9353-840bd249d790" (UID: "1d34e5e5-88b0-4040-9353-840bd249d790"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:58:55.972314 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.972277    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbac47c-533a-4e63-b65b-c87a2a6eee14" path="/var/lib/kubelet/pods/6dbac47c-533a-4e63-b65b-c87a2a6eee14/volumes"
Nov 03 19:58:55.981549 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.981517    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.981549 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.981551    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d34e5e5-88b0-4040-9353-840bd249d790-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.981825 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.981568    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d34e5e5-88b0-4040-9353-840bd249d790-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:55.981825 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:55.981583    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bddwl\" (UniqueName: \"kubernetes.io/projected/1d34e5e5-88b0-4040-9353-840bd249d790-kube-api-access-bddwl\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:56.731520 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:56.731471    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"
Nov 03 19:58:56.732020 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:56.731517    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm" event={"ID":"1d34e5e5-88b0-4040-9353-840bd249d790","Type":"ContainerDied","Data":"20ad5b2695ed1a6358e5f8612160076c693e5654653c1877b35a93f137821bae"}
Nov 03 19:58:56.732020 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:56.731583    5659 scope.go:122] "RemoveContainer" containerID="21abac9cea58740d3e74430a303aae277c5870c67bf42a66de51adc0dfb8ee2a"
Nov 03 19:58:56.732020 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:56.731764    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"
Nov 03 19:58:56.735209 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:56.735185    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:56.756186 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:56.755942    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"]
Nov 03 19:58:56.763325 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:56.763288    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-t2nx2"]
Nov 03 19:58:56.816505 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:56.815559    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"]
Nov 03 19:58:56.826307 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:56.826267    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5p7dm"]
Nov 03 19:58:57.832840 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:57.832807    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:57.972459 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:57.972423    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dde3468-dceb-4d4e-8e71-1eba7e214145" path="/var/lib/kubelet/pods/0dde3468-dceb-4d4e-8e71-1eba7e214145/volumes"
Nov 03 19:58:57.972950 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:57.972933    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" path="/var/lib/kubelet/pods/1d34e5e5-88b0-4040-9353-840bd249d790/volumes"
Nov 03 19:58:58.072990 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:58.072961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:58.485565 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:58.485511    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"]
Nov 03 19:58:58.485776 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:58.485750    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g" podUID="53360c4d-e89f-45e5-9b77-4df72a1f93d0" containerName="maas-api" containerID="cri-o://07db43dd8651de1032ed41e7a7ea0fd2fde3085a99a813e453c0956d396056af" gracePeriod=30
Nov 03 19:58:58.596789 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:58.596752    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:58.949218 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:58.949128    5659 generic.go:356] "Generic (PLEG): container finished" podID="53360c4d-e89f-45e5-9b77-4df72a1f93d0" containerID="07db43dd8651de1032ed41e7a7ea0fd2fde3085a99a813e453c0956d396056af" exitCode=0
Nov 03 19:58:58.949686 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:58.949297    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g" event={"ID":"53360c4d-e89f-45e5-9b77-4df72a1f93d0","Type":"ContainerDied","Data":"07db43dd8651de1032ed41e7a7ea0fd2fde3085a99a813e453c0956d396056af"}
Nov 03 19:58:59.032964 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.032006    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:59.049087 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.048464    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:58:59.102387 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.102353    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-maas-api-tls\") pod \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") "
Nov 03 19:58:59.102594 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.102427    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53360c4d-e89f-45e5-9b77-4df72a1f93d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53360c4d-e89f-45e5-9b77-4df72a1f93d0-openshift-service-ca\") pod \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") "
Nov 03 19:58:59.102704 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.102649    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-metrics-certs\") pod \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") "
Nov 03 19:58:59.102704 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.102692    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53360c4d-e89f-45e5-9b77-4df72a1f93d0-kube-api-access-pd2tz\" (UniqueName: \"kubernetes.io/projected/53360c4d-e89f-45e5-9b77-4df72a1f93d0-kube-api-access-pd2tz\") pod \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\" (UID: \"53360c4d-e89f-45e5-9b77-4df72a1f93d0\") "
Nov 03 19:58:59.104952 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.104392    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53360c4d-e89f-45e5-9b77-4df72a1f93d0-openshift-service-ca" pod "53360c4d-e89f-45e5-9b77-4df72a1f93d0" (UID: "53360c4d-e89f-45e5-9b77-4df72a1f93d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:59.114764 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.114706    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-maas-api-tls" pod "53360c4d-e89f-45e5-9b77-4df72a1f93d0" (UID: "53360c4d-e89f-45e5-9b77-4df72a1f93d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:58:59.114764 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.114706    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53360c4d-e89f-45e5-9b77-4df72a1f93d0-kube-api-access-pd2tz" pod "53360c4d-e89f-45e5-9b77-4df72a1f93d0" (UID: "53360c4d-e89f-45e5-9b77-4df72a1f93d0"). InnerVolumeSpecName "kube-api-access-pd2tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:59.115148 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.115121    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-metrics-certs" pod "53360c4d-e89f-45e5-9b77-4df72a1f93d0" (UID: "53360c4d-e89f-45e5-9b77-4df72a1f93d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:58:59.206526 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.206472    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:59.206526 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.206530    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd2tz\" (UniqueName: \"kubernetes.io/projected/53360c4d-e89f-45e5-9b77-4df72a1f93d0-kube-api-access-pd2tz\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:59.206798 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.206549    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53360c4d-e89f-45e5-9b77-4df72a1f93d0-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:59.206798 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.206564    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53360c4d-e89f-45e5-9b77-4df72a1f93d0-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:59.961181 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.961146    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"
Nov 03 19:58:59.968661 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.968633    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g" event={"ID":"53360c4d-e89f-45e5-9b77-4df72a1f93d0","Type":"ContainerDied","Data":"17b88166c4a7fa4ac0161305555730dc126ac516573835654763289aec6f0bce"}
Nov 03 19:58:59.968826 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.968687    5659 scope.go:122] "RemoveContainer" containerID="07db43dd8651de1032ed41e7a7ea0fd2fde3085a99a813e453c0956d396056af"
Nov 03 19:58:59.993273 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.993247    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"]
Nov 03 19:58:59.998288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:58:59.998264    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-29f633-67f86bfb74-mds9g"]
Nov 03 19:59:00.100776 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:00.100749    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:01.034209 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:01.034175    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:01.925205 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:01.925173    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:01.972536 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:01.972083    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53360c4d-e89f-45e5-9b77-4df72a1f93d0" path="/var/lib/kubelet/pods/53360c4d-e89f-45e5-9b77-4df72a1f93d0/volumes"
Nov 03 19:59:02.724606 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:02.724574    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:03.516666 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:03.516632    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:04.246244 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.246196    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"]
Nov 03 19:59:04.247724 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.247693    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6dbac47c-533a-4e63-b65b-c87a2a6eee14" containerName="payload-pre-processing"
Nov 03 19:59:04.247911 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.247893    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbac47c-533a-4e63-b65b-c87a2a6eee14" containerName="payload-pre-processing"
Nov 03 19:59:04.248040 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.248026    5659 cpu_manager.go:414] "containerMap: removing container" podUID="0dde3468-dceb-4d4e-8e71-1eba7e214145" containerName="istio-proxy"
Nov 03 19:59:04.248130 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.248119    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dde3468-dceb-4d4e-8e71-1eba7e214145" containerName="istio-proxy"
Nov 03 19:59:04.248228 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.248216    5659 cpu_manager.go:414] "containerMap: removing container" podUID="30199a75-1c40-43e5-9c62-a279c19890d4" containerName="payload-processing"
Nov 03 19:59:04.248370 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.248354    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30199a75-1c40-43e5-9c62-a279c19890d4" containerName="payload-processing"
Nov 03 19:59:04.248547 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.248533    5659 cpu_manager.go:414] "containerMap: removing container" podUID="670fc60a-c585-4c83-a527-143c4c951f89" containerName="payload-processing"
Nov 03 19:59:04.248653 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.248641    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670fc60a-c585-4c83-a527-143c4c951f89" containerName="payload-processing"
Nov 03 19:59:04.249397 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.249370    5659 cpu_manager.go:414] "containerMap: removing container" podUID="53360c4d-e89f-45e5-9b77-4df72a1f93d0" containerName="maas-api"
Nov 03 19:59:04.249397 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.249392    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53360c4d-e89f-45e5-9b77-4df72a1f93d0" containerName="maas-api"
Nov 03 19:59:04.249684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.249418    5659 cpu_manager.go:414] "containerMap: removing container" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" containerName="istio-proxy"
Nov 03 19:59:04.249684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.249426    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" containerName="istio-proxy"
Nov 03 19:59:04.249684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.249624    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d34e5e5-88b0-4040-9353-840bd249d790" containerName="istio-proxy"
Nov 03 19:59:04.249684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.249642    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbac47c-533a-4e63-b65b-c87a2a6eee14" containerName="payload-pre-processing"
Nov 03 19:59:04.249684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.249654    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dde3468-dceb-4d4e-8e71-1eba7e214145" containerName="istio-proxy"
Nov 03 19:59:04.249684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.249664    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="30199a75-1c40-43e5-9c62-a279c19890d4" containerName="payload-processing"
Nov 03 19:59:04.249684 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.249683    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="670fc60a-c585-4c83-a527-143c4c951f89" containerName="payload-processing"
Nov 03 19:59:04.250045 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.249693    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="53360c4d-e89f-45e5-9b77-4df72a1f93d0" containerName="maas-api"
Nov 03 19:59:04.260368 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.260337    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"]
Nov 03 19:59:04.260556 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.260458    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.268158 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.265707    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3faa07-serving-cert\""
Nov 03 19:59:04.284636 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.284600    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-metrics-certs\") pod \"maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.284836 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.284649    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd5gs\" (UniqueName: \"kubernetes.io/projected/56c05f1f-b29e-4346-b5e3-69381c6b626f-kube-api-access-hd5gs\") pod \"maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.284894 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.284829    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c05f1f-b29e-4346-b5e3-69381c6b626f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.285017 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.284996    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.288935 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.288903    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:04.386992 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.386952    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.387172 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.387086    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-metrics-certs\") pod \"maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.387172 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.387108    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd5gs\" (UniqueName: \"kubernetes.io/projected/56c05f1f-b29e-4346-b5e3-69381c6b626f-kube-api-access-hd5gs\") pod \"maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.387172 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.387149    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c05f1f-b29e-4346-b5e3-69381c6b626f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.387872 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.387850    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c05f1f-b29e-4346-b5e3-69381c6b626f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.389763 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.389736    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.389972 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.389946    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-metrics-certs\") pod \"maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.394278 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.394249    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"]
Nov 03 19:59:04.398524 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.398505    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.407828 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.407789    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd5gs\" (UniqueName: \"kubernetes.io/projected/56c05f1f-b29e-4346-b5e3-69381c6b626f-kube-api-access-hd5gs\") pod \"maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.412822 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.412789    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"]
Nov 03 19:59:04.472507 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.458681    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:04.489034 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.488995    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.489282 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.489043    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.489282 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.489087    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7280958a-874a-44e5-8360-283eeff83728-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.489282 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.489130    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.489282 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.489162    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.489282 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.489236    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7280958a-874a-44e5-8360-283eeff83728-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.489657 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.489357    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsjpn\" (UniqueName: \"kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-kube-api-access-dsjpn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.580770 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.580674    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:04.592853 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.592805    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:04.593552 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.593514    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.593675 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.593617    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7280958a-874a-44e5-8360-283eeff83728-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.593754 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.593735    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsjpn\" (UniqueName: \"kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-kube-api-access-dsjpn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.593932 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.593907    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.593992 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.593944    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.593992 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.593991    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7280958a-874a-44e5-8360-283eeff83728-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.594107 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.594039    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.612500 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.600803    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7280958a-874a-44e5-8360-283eeff83728-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.612500 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.605432    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.612500 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.605917    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.612500 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.611238    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7280958a-874a-44e5-8360-283eeff83728-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.618087 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.613042    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.618087 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.616586    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.618753 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.618719    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsjpn\" (UniqueName: \"kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-kube-api-access-dsjpn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rcpkf\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.776952 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.768465    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:04.836383 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:04.836238    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:05.063288 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:05.060756    5659 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:59:05.069067 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:05.068922    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"]
Nov 03 19:59:05.180858 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:05.180466    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"]
Nov 03 19:59:05.181542 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:05.181475    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:59:05.181648 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:05.181570    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:59:05.206839 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:05.206816    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:05.942852 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:05.942739    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:06.094060 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:06.094023    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh" event={"ID":"56c05f1f-b29e-4346-b5e3-69381c6b626f","Type":"ContainerStarted","Data":"32bc1089ca279406104de34cf1f0cda5cc9d2dd93a0e218efb92b8b59184568e"}
Nov 03 19:59:06.094314 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:06.094295    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh" event={"ID":"56c05f1f-b29e-4346-b5e3-69381c6b626f","Type":"ContainerStarted","Data":"bca232cca06d1cc2fa82fd3249bb487731cfc54a9efd25d8f96aaa2f34740255"}
Nov 03 19:59:06.095528 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:06.095508    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:06.100752 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:06.098688    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf" event={"ID":"7280958a-874a-44e5-8360-283eeff83728","Type":"ContainerStarted","Data":"06067807e3b04b8414242656fc57da5149a8e725a39a13948583d6d96664828e"}
Nov 03 19:59:06.100752 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:06.098733    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf" event={"ID":"7280958a-874a-44e5-8360-283eeff83728","Type":"ContainerStarted","Data":"e09d212107f0763f47068a655b6bda8e43f942315a6e27697402fec6cd7d1f63"}
Nov 03 19:59:06.100752 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:06.099842    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:06.101671 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:06.101648    5659 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rcpkf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:59:06.101820 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:06.101700    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf" podUID="7280958a-874a-44e5-8360-283eeff83728" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:59:06.121868 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:06.120029    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh" podStartSLOduration=1.510271381 podStartE2EDuration="2.120010887s" totalImagesPullingTime="609.739506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:06.119783437 +0000 UTC m=+1298.274685853" watchObservedRunningTime="2026-11-03 19:59:06.120010887 +0000 UTC m=+1298.274913304"
Nov 03 19:59:06.149124 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:06.149064    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf" podStartSLOduration=2.149044685 podStartE2EDuration="2.149044685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:06.144004425 +0000 UTC m=+1298.298906872" watchObservedRunningTime="2026-11-03 19:59:06.149044685 +0000 UTC m=+1298.303947105"
Nov 03 19:59:06.159925 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:06.159868    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:06.753543 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:06.751245    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:06.898536 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:06.895309    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:07.106231 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:07.106155    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 19:59:07.206453 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:07.206421    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:08.297513 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:08.297453    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:08.490054 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:08.490023    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:09.501438 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:09.501403    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:10.557512 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:10.557448    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:10.771844 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:10.771798    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:11.669680 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:11.669646    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:11.850884 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:11.850843    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:12.055718 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:12.055633    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:12.799925 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:12.799879    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:12.935419 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:12.935369    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:13.133551 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:13.133456    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 19:59:13.729401 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:13.729373    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:14.523351 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:14.523311    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:14.653700 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:14.653667    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:14.793431 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:14.793343    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:15.221225 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:59:15.221182    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:59:15.657237 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:15.657126    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:16.365850 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:16.365815    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:17.177254 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:17.177208    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:17.974998 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:17.974961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:18.970777 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:18.970741    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:19.639757 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:19.639722    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:20.366006 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:20.365957    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:21.227195 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:21.227146    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:21.359688 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:21.359635    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:21.581164 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:21.581078    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:22.248261 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:22.248224    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:22.825056 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:22.825015    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:22.960904 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:22.960864    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:23.579358 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:23.579325    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:24.513548 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:24.513518    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:25.447371 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:25.447331    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:26.357283 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:26.357232    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:27.254081 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:27.254045    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:28.161742 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:28.161682    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:28.774167 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:28.774127    5659 scope.go:122] "RemoveContainer" containerID="ee578c20e6a5cfbaebcede3bd07d17398fb12d4d1999375500d3b9ef87e29c1e"
Nov 03 19:59:28.938346 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:28.938306    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:29.630463 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:29.630425    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:30.335261 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:30.335210    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:31.119346 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:31.119301    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:31.553248 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:31.553176    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:31.735909 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:31.735875    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:32.431848 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:32.431814    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:32.566313 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:32.566281    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:32.723188 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:32.723148    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:33.117076 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:33.116987    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:33.589794 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:33.589316    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:33.790426 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:33.790392    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:34.855513 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:34.855467    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:35.722974 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:35.722940    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:35.851289 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:35.851212    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:35.996770 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:35.996692    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:36.132124 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:36.132087    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:36.275445 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:36.275307    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:36.520921 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:36.520887    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:37.241519 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:37.241467    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:37.373130 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:37.373097    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:37.525012 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:37.524927    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:37.660007 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:37.659973    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:38.222308 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:38.222275    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:38.410186 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:38.410155    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:38.543717 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:38.543641    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:39.320868 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:39.320835    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:40.127013 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:40.126976    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:40.708697 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:40.708665    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:41.050992 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:41.050882    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:41.718322 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:41.718293    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:42.559845 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:42.559809    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:43.256100 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:43.256063    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:44.134967 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:44.134933    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:44.824593 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:44.824555    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:45.217886 ip-10-0-133-249 kubenswrapper[5659]: E1103 19:59:45.217846    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 19:59:45.503824 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:45.503732    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:46.530847 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:46.530815    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:47.081554 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:47.081439    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:47.220610 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:47.220577    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:47.733854 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:47.733819    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:48.026788 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:48.026705    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:49.038497 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:49.038452    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:49.168219 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:49.168186    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:49.708859 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:49.708827    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:50.145702 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:50.145613    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:50.960939 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:50.960896    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:51.747633 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:51.747593    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:52.546144 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:52.546112    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:53.321792 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:53.314391    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:53.481062 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:53.481026    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:53.915668 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:53.915638    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:54.212434 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:54.212401    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:54.360471 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:54.360440    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:54.820824 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:54.820772    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:54.953705 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:54.953661    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:55.211253 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:55.211217    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:55.754001 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:55.753968    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:56.028694 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:56.028616    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:56.456081 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:56.456052    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:57.166237 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:57.166195    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:57.840777 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:57.840743    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:58.532797 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:58.532767    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:59.342677 ip-10-0-133-249 kubenswrapper[5659]: I1103 19:59:59.342621    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:00.166722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.166309    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"]
Nov 03 20:00:00.174739 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.172063    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"]
Nov 03 20:00:00.174739 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.174711    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"
Nov 03 20:00:00.188507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.181789    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"]
Nov 03 20:00:00.188507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.186719    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"
Nov 03 20:00:00.188507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.186968    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"]
Nov 03 20:00:00.188507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.188315    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"
Nov 03 20:00:00.194053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.194026    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"]
Nov 03 20:00:00.196570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.196547    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"]
Nov 03 20:00:00.201065 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.201044    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"]
Nov 03 20:00:00.206420 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.206391    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"]
Nov 03 20:00:00.206752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.206643    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"
Nov 03 20:00:00.211071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.211054    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"
Nov 03 20:00:00.224999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.224972    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"]
Nov 03 20:00:00.227119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.227098    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"]
Nov 03 20:00:00.263043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.263024    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\""
Nov 03 20:00:00.352300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.352269    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lq2d\" (UniqueName: \"kubernetes.io/projected/8027f464-332c-430c-976d-22327de2da7a-kube-api-access-9lq2d\") pod \"maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj\" (UID: \"8027f464-332c-430c-976d-22327de2da7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"
Nov 03 20:00:00.352518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.352335    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f77ab6-e954-4124-bd4f-ea0891241124-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx\" (UID: \"33f77ab6-e954-4124-bd4f-ea0891241124\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"
Nov 03 20:00:00.352518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.352420    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4bde4d-2410-4104-abaf-edc03cf5e895-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl\" (UID: \"3f4bde4d-2410-4104-abaf-edc03cf5e895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"
Nov 03 20:00:00.352518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.352510    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr\" (UID: \"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"
Nov 03 20:00:00.352694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.352556    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d32f4c-445e-4aa4-8ecb-bacb6903c699-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895600-tpg7b\" (UID: \"64d32f4c-445e-4aa4-8ecb-bacb6903c699\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"
Nov 03 20:00:00.352694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.352575    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn6p2\" (UniqueName: \"kubernetes.io/projected/33f77ab6-e954-4124-bd4f-ea0891241124-kube-api-access-wn6p2\") pod \"maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx\" (UID: \"33f77ab6-e954-4124-bd4f-ea0891241124\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"
Nov 03 20:00:00.352694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.352608    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz86s\" (UniqueName: \"kubernetes.io/projected/3f4bde4d-2410-4104-abaf-edc03cf5e895-kube-api-access-hz86s\") pod \"maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl\" (UID: \"3f4bde4d-2410-4104-abaf-edc03cf5e895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"
Nov 03 20:00:00.352855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.352750    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qsg2\" (UniqueName: \"kubernetes.io/projected/64d32f4c-445e-4aa4-8ecb-bacb6903c699-kube-api-access-5qsg2\") pod \"maas-api-key-cleanup-29895600-tpg7b\" (UID: \"64d32f4c-445e-4aa4-8ecb-bacb6903c699\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"
Nov 03 20:00:00.352855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.352788    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56d52\" (UniqueName: \"kubernetes.io/projected/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-kube-api-access-56d52\") pod \"maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr\" (UID: \"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"
Nov 03 20:00:00.352960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.352902    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8027f464-332c-430c-976d-22327de2da7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj\" (UID: \"8027f464-332c-430c-976d-22327de2da7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"
Nov 03 20:00:00.431858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.431782    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:00.467515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.466767    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qsg2\" (UniqueName: \"kubernetes.io/projected/64d32f4c-445e-4aa4-8ecb-bacb6903c699-kube-api-access-5qsg2\") pod \"maas-api-key-cleanup-29895600-tpg7b\" (UID: \"64d32f4c-445e-4aa4-8ecb-bacb6903c699\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"
Nov 03 20:00:00.467515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.466836    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56d52\" (UniqueName: \"kubernetes.io/projected/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-kube-api-access-56d52\") pod \"maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr\" (UID: \"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"
Nov 03 20:00:00.467515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.466962    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8027f464-332c-430c-976d-22327de2da7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj\" (UID: \"8027f464-332c-430c-976d-22327de2da7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"
Nov 03 20:00:00.467515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.467044    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lq2d\" (UniqueName: \"kubernetes.io/projected/8027f464-332c-430c-976d-22327de2da7a-kube-api-access-9lq2d\") pod \"maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj\" (UID: \"8027f464-332c-430c-976d-22327de2da7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"
Nov 03 20:00:00.491528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.482796    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8027f464-332c-430c-976d-22327de2da7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj\" (UID: \"8027f464-332c-430c-976d-22327de2da7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"
Nov 03 20:00:00.491528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.483057    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f77ab6-e954-4124-bd4f-ea0891241124-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx\" (UID: \"33f77ab6-e954-4124-bd4f-ea0891241124\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"
Nov 03 20:00:00.491528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.483128    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4bde4d-2410-4104-abaf-edc03cf5e895-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl\" (UID: \"3f4bde4d-2410-4104-abaf-edc03cf5e895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"
Nov 03 20:00:00.491528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.483261    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr\" (UID: \"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"
Nov 03 20:00:00.491528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.483301    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d32f4c-445e-4aa4-8ecb-bacb6903c699-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895600-tpg7b\" (UID: \"64d32f4c-445e-4aa4-8ecb-bacb6903c699\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"
Nov 03 20:00:00.491528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.483444    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6p2\" (UniqueName: \"kubernetes.io/projected/33f77ab6-e954-4124-bd4f-ea0891241124-kube-api-access-wn6p2\") pod \"maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx\" (UID: \"33f77ab6-e954-4124-bd4f-ea0891241124\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"
Nov 03 20:00:00.491528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.483499    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz86s\" (UniqueName: \"kubernetes.io/projected/3f4bde4d-2410-4104-abaf-edc03cf5e895-kube-api-access-hz86s\") pod \"maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl\" (UID: \"3f4bde4d-2410-4104-abaf-edc03cf5e895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"
Nov 03 20:00:00.491528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.484666    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4bde4d-2410-4104-abaf-edc03cf5e895-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl\" (UID: \"3f4bde4d-2410-4104-abaf-edc03cf5e895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"
Nov 03 20:00:00.491528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.485308    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr\" (UID: \"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"
Nov 03 20:00:00.491528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.487770    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d32f4c-445e-4aa4-8ecb-bacb6903c699-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895600-tpg7b\" (UID: \"64d32f4c-445e-4aa4-8ecb-bacb6903c699\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"
Nov 03 20:00:00.491528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.488591    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f77ab6-e954-4124-bd4f-ea0891241124-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx\" (UID: \"33f77ab6-e954-4124-bd4f-ea0891241124\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"
Nov 03 20:00:00.507758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.507719    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qsg2\" (UniqueName: \"kubernetes.io/projected/64d32f4c-445e-4aa4-8ecb-bacb6903c699-kube-api-access-5qsg2\") pod \"maas-api-key-cleanup-29895600-tpg7b\" (UID: \"64d32f4c-445e-4aa4-8ecb-bacb6903c699\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"
Nov 03 20:00:00.511157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.511093    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lq2d\" (UniqueName: \"kubernetes.io/projected/8027f464-332c-430c-976d-22327de2da7a-kube-api-access-9lq2d\") pod \"maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj\" (UID: \"8027f464-332c-430c-976d-22327de2da7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"
Nov 03 20:00:00.525079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.525002    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56d52\" (UniqueName: \"kubernetes.io/projected/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-kube-api-access-56d52\") pod \"maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr\" (UID: \"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"
Nov 03 20:00:00.525079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.525020    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6p2\" (UniqueName: \"kubernetes.io/projected/33f77ab6-e954-4124-bd4f-ea0891241124-kube-api-access-wn6p2\") pod \"maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx\" (UID: \"33f77ab6-e954-4124-bd4f-ea0891241124\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"
Nov 03 20:00:00.527100 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.527043    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz86s\" (UniqueName: \"kubernetes.io/projected/3f4bde4d-2410-4104-abaf-edc03cf5e895-kube-api-access-hz86s\") pod \"maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl\" (UID: \"3f4bde4d-2410-4104-abaf-edc03cf5e895\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"
Nov 03 20:00:00.529766 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.529738    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"
Nov 03 20:00:00.544521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.543380    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"
Nov 03 20:00:00.565626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.565596    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"
Nov 03 20:00:00.567777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.567728    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"
Nov 03 20:00:00.577961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.577936    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"
Nov 03 20:00:00.857425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.857357    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"]
Nov 03 20:00:00.905862 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:00:00.905829    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d32f4c_445e_4aa4_8ecb_bacb6903c699.slice/crio-ed759b344ad373d563effa1a1cd5f9d2dbbb286be7b3ab97ef5d7c3dd5b0a524 WatchSource:0}: Error finding container ed759b344ad373d563effa1a1cd5f9d2dbbb286be7b3ab97ef5d7c3dd5b0a524: Status 404 returned error can't find the container with id ed759b344ad373d563effa1a1cd5f9d2dbbb286be7b3ab97ef5d7c3dd5b0a524
Nov 03 20:00:00.906025 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.906004    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"]
Nov 03 20:00:00.933307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.933280    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"]
Nov 03 20:00:00.940192 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:00:00.940164    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4bde4d_2410_4104_abaf_edc03cf5e895.slice/crio-c0b4e8a8781ff7cab3b07e527575c4f783aa770a8249e7b55662d13865d8cc78 WatchSource:0}: Error finding container c0b4e8a8781ff7cab3b07e527575c4f783aa770a8249e7b55662d13865d8cc78: Status 404 returned error can't find the container with id c0b4e8a8781ff7cab3b07e527575c4f783aa770a8249e7b55662d13865d8cc78
Nov 03 20:00:00.944567 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:00:00.944540    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f77ab6_e954_4124_bd4f_ea0891241124.slice/crio-5a108b0086993addec92c4f22c1092ec090a388c9aa53346db2dd1560fed65ee WatchSource:0}: Error finding container 5a108b0086993addec92c4f22c1092ec090a388c9aa53346db2dd1560fed65ee: Status 404 returned error can't find the container with id 5a108b0086993addec92c4f22c1092ec090a388c9aa53346db2dd1560fed65ee
Nov 03 20:00:00.945255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.945234    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"]
Nov 03 20:00:00.975831 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:00.975809    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"]
Nov 03 20:00:00.981042 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:00:00.981014    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8027f464_332c_430c_976d_22327de2da7a.slice/crio-e78761f8d06b505f043b217c60514f646c58f811ee69a43912978a562e5f8cc9 WatchSource:0}: Error finding container e78761f8d06b505f043b217c60514f646c58f811ee69a43912978a562e5f8cc9: Status 404 returned error can't find the container with id e78761f8d06b505f043b217c60514f646c58f811ee69a43912978a562e5f8cc9
Nov 03 20:00:01.431357 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.431324    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:01.694067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.694030    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl" event={"ID":"3f4bde4d-2410-4104-abaf-edc03cf5e895","Type":"ContainerStarted","Data":"93bd37242c2351475639f5aac98be75387f56a358f78caa5692e7f2095bc1dbd"}
Nov 03 20:00:01.694248 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.694122    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl" event={"ID":"3f4bde4d-2410-4104-abaf-edc03cf5e895","Type":"ContainerStarted","Data":"c0b4e8a8781ff7cab3b07e527575c4f783aa770a8249e7b55662d13865d8cc78"}
Nov 03 20:00:01.695957 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.695929    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr" event={"ID":"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c","Type":"ContainerStarted","Data":"f9a4dfadefdfcb0b1847dbfc3ac78629077deb535d1398fd1d76167ef051efc2"}
Nov 03 20:00:01.696097 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.695978    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr" event={"ID":"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c","Type":"ContainerStarted","Data":"39c78a3a5c668850a25e9f185042bd12178227520ef02f6861c93072ffd96ffe"}
Nov 03 20:00:01.697651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.697618    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx" event={"ID":"33f77ab6-e954-4124-bd4f-ea0891241124","Type":"ContainerStarted","Data":"702965e2ecf9eb41987b860c5168f64d2b651bebb639a627c71293bc3f094c36"}
Nov 03 20:00:01.697783 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.697664    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx" event={"ID":"33f77ab6-e954-4124-bd4f-ea0891241124","Type":"ContainerStarted","Data":"5a108b0086993addec92c4f22c1092ec090a388c9aa53346db2dd1560fed65ee"}
Nov 03 20:00:01.699274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.699248    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b" event={"ID":"64d32f4c-445e-4aa4-8ecb-bacb6903c699","Type":"ContainerStarted","Data":"ae5cf7a6d3c6a953ef388918a713ef6239bd603a94ba2a12f92cad983533e8bc"}
Nov 03 20:00:01.699399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.699298    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b" event={"ID":"64d32f4c-445e-4aa4-8ecb-bacb6903c699","Type":"ContainerStarted","Data":"ed759b344ad373d563effa1a1cd5f9d2dbbb286be7b3ab97ef5d7c3dd5b0a524"}
Nov 03 20:00:01.700948 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.700927    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj" event={"ID":"8027f464-332c-430c-976d-22327de2da7a","Type":"ContainerStarted","Data":"3e727b9fc22e9724c6b27d505ae1e3bf316c35021c3677691ddd0cfcc24abc43"}
Nov 03 20:00:01.701042 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.700962    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj" event={"ID":"8027f464-332c-430c-976d-22327de2da7a","Type":"ContainerStarted","Data":"e78761f8d06b505f043b217c60514f646c58f811ee69a43912978a562e5f8cc9"}
Nov 03 20:00:01.714733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.714681    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl" podStartSLOduration=1.714665487 podStartE2EDuration="1.714665487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:01.710804483 +0000 UTC m=+1353.865706902" watchObservedRunningTime="2026-11-03 20:00:01.714665487 +0000 UTC m=+1353.869567906"
Nov 03 20:00:01.735506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.735163    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj" podStartSLOduration=1.7351419959999999 podStartE2EDuration="1.735141996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:01.731424153 +0000 UTC m=+1353.886326588" watchObservedRunningTime="2026-11-03 20:00:01.735141996 +0000 UTC m=+1353.890044416"
Nov 03 20:00:01.755599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.755435    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr" podStartSLOduration=1.755416281 podStartE2EDuration="1.755416281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:01.752475435 +0000 UTC m=+1353.907377855" watchObservedRunningTime="2026-11-03 20:00:01.755416281 +0000 UTC m=+1353.910318702"
Nov 03 20:00:01.774770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.774707    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b" podStartSLOduration=1.774692753 podStartE2EDuration="1.774692753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:01.770647416 +0000 UTC m=+1353.925549829" watchObservedRunningTime="2026-11-03 20:00:01.774692753 +0000 UTC m=+1353.929595171"
Nov 03 20:00:01.797070 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:01.797001    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx" podStartSLOduration=1.7969802000000001 podStartE2EDuration="1.7969802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:01.796265192 +0000 UTC m=+1353.951167612" watchObservedRunningTime="2026-11-03 20:00:01.7969802 +0000 UTC m=+1353.951882624"
Nov 03 20:00:02.268377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.268336    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:02.633399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.633360    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:02.709571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.709534    5659 generic.go:356] "Generic (PLEG): container finished" podID="64d32f4c-445e-4aa4-8ecb-bacb6903c699" containerID="ae5cf7a6d3c6a953ef388918a713ef6239bd603a94ba2a12f92cad983533e8bc" exitCode=60
Nov 03 20:00:02.709753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.709615    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b" event={"ID":"64d32f4c-445e-4aa4-8ecb-bacb6903c699","Type":"ContainerDied","Data":"ae5cf7a6d3c6a953ef388918a713ef6239bd603a94ba2a12f92cad983533e8bc"}
Nov 03 20:00:02.709966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.709948    5659 scope.go:122] "RemoveContainer" containerID="ae5cf7a6d3c6a953ef388918a713ef6239bd603a94ba2a12f92cad983533e8bc"
Nov 03 20:00:02.711304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.711279    5659 generic.go:356] "Generic (PLEG): container finished" podID="8027f464-332c-430c-976d-22327de2da7a" containerID="3e727b9fc22e9724c6b27d505ae1e3bf316c35021c3677691ddd0cfcc24abc43" exitCode=60
Nov 03 20:00:02.711429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.711408    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj" event={"ID":"8027f464-332c-430c-976d-22327de2da7a","Type":"ContainerDied","Data":"3e727b9fc22e9724c6b27d505ae1e3bf316c35021c3677691ddd0cfcc24abc43"}
Nov 03 20:00:02.711654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.711639    5659 scope.go:122] "RemoveContainer" containerID="3e727b9fc22e9724c6b27d505ae1e3bf316c35021c3677691ddd0cfcc24abc43"
Nov 03 20:00:02.713276 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.713179    5659 generic.go:356] "Generic (PLEG): container finished" podID="3f4bde4d-2410-4104-abaf-edc03cf5e895" containerID="93bd37242c2351475639f5aac98be75387f56a358f78caa5692e7f2095bc1dbd" exitCode=60
Nov 03 20:00:02.713276 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.713244    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl" event={"ID":"3f4bde4d-2410-4104-abaf-edc03cf5e895","Type":"ContainerDied","Data":"93bd37242c2351475639f5aac98be75387f56a358f78caa5692e7f2095bc1dbd"}
Nov 03 20:00:02.713535 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.713476    5659 scope.go:122] "RemoveContainer" containerID="93bd37242c2351475639f5aac98be75387f56a358f78caa5692e7f2095bc1dbd"
Nov 03 20:00:02.715097 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.715066    5659 generic.go:356] "Generic (PLEG): container finished" podID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" containerID="f9a4dfadefdfcb0b1847dbfc3ac78629077deb535d1398fd1d76167ef051efc2" exitCode=60
Nov 03 20:00:02.715239 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.715208    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr" event={"ID":"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c","Type":"ContainerDied","Data":"f9a4dfadefdfcb0b1847dbfc3ac78629077deb535d1398fd1d76167ef051efc2"}
Nov 03 20:00:02.715433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.715412    5659 scope.go:122] "RemoveContainer" containerID="f9a4dfadefdfcb0b1847dbfc3ac78629077deb535d1398fd1d76167ef051efc2"
Nov 03 20:00:02.716964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.716945    5659 generic.go:356] "Generic (PLEG): container finished" podID="33f77ab6-e954-4124-bd4f-ea0891241124" containerID="702965e2ecf9eb41987b860c5168f64d2b651bebb639a627c71293bc3f094c36" exitCode=60
Nov 03 20:00:02.717075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.716972    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx" event={"ID":"33f77ab6-e954-4124-bd4f-ea0891241124","Type":"ContainerDied","Data":"702965e2ecf9eb41987b860c5168f64d2b651bebb639a627c71293bc3f094c36"}
Nov 03 20:00:02.717218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:02.717203    5659 scope.go:122] "RemoveContainer" containerID="702965e2ecf9eb41987b860c5168f64d2b651bebb639a627c71293bc3f094c36"
Nov 03 20:00:03.097322 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:03.097291    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:03.402319 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:03.402285    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:03.550574 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:03.550433    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:03.726207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:03.726172    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl" event={"ID":"3f4bde4d-2410-4104-abaf-edc03cf5e895","Type":"ContainerStarted","Data":"d0492b00c6052cc0636c9a5c087a81d389e9b14359e72138fac2554f9853ebd9"}
Nov 03 20:00:03.728262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:03.728236    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr" event={"ID":"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c","Type":"ContainerStarted","Data":"892bc066ff365f7d03884d112171321aa29d16ffe1508bde31d995da3d285941"}
Nov 03 20:00:03.730271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:03.730249    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx" event={"ID":"33f77ab6-e954-4124-bd4f-ea0891241124","Type":"ContainerStarted","Data":"b83e152f83f7361f8d73078580580114af33e664c6f602e4b95bb2f9ba943c2a"}
Nov 03 20:00:03.737609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:03.735278    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b" event={"ID":"64d32f4c-445e-4aa4-8ecb-bacb6903c699","Type":"ContainerStarted","Data":"9e3d884bdf9a200ce49271427c0cf90ec694d00053d1a07f79822e31a8eb39a8"}
Nov 03 20:00:03.745143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:03.742588    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj" event={"ID":"8027f464-332c-430c-976d-22327de2da7a","Type":"ContainerStarted","Data":"cda7952bf5321b815035c52c83971caab8d48697b9c4c1fa4a2b9d8add3cfe4f"}
Nov 03 20:00:04.191294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.191200    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:04.432059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.432017    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:04.757272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.757234    5659 generic.go:356] "Generic (PLEG): container finished" podID="64d32f4c-445e-4aa4-8ecb-bacb6903c699" containerID="9e3d884bdf9a200ce49271427c0cf90ec694d00053d1a07f79822e31a8eb39a8" exitCode=60
Nov 03 20:00:04.757973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.757944    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b" event={"ID":"64d32f4c-445e-4aa4-8ecb-bacb6903c699","Type":"ContainerDied","Data":"9e3d884bdf9a200ce49271427c0cf90ec694d00053d1a07f79822e31a8eb39a8"}
Nov 03 20:00:04.758091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.758044    5659 scope.go:122] "RemoveContainer" containerID="ae5cf7a6d3c6a953ef388918a713ef6239bd603a94ba2a12f92cad983533e8bc"
Nov 03 20:00:04.758648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.758476    5659 scope.go:122] "RemoveContainer" containerID="9e3d884bdf9a200ce49271427c0cf90ec694d00053d1a07f79822e31a8eb39a8"
Nov 03 20:00:04.758898 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:04.758876    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895600-tpg7b_odh-ai-gateway-infra(64d32f4c-445e-4aa4-8ecb-bacb6903c699)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b" podUID="64d32f4c-445e-4aa4-8ecb-bacb6903c699"
Nov 03 20:00:04.774259 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.774223    5659 generic.go:356] "Generic (PLEG): container finished" podID="8027f464-332c-430c-976d-22327de2da7a" containerID="cda7952bf5321b815035c52c83971caab8d48697b9c4c1fa4a2b9d8add3cfe4f" exitCode=60
Nov 03 20:00:04.775676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.775650    5659 scope.go:122] "RemoveContainer" containerID="cda7952bf5321b815035c52c83971caab8d48697b9c4c1fa4a2b9d8add3cfe4f"
Nov 03 20:00:04.775829 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.775707    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj" event={"ID":"8027f464-332c-430c-976d-22327de2da7a","Type":"ContainerDied","Data":"cda7952bf5321b815035c52c83971caab8d48697b9c4c1fa4a2b9d8add3cfe4f"}
Nov 03 20:00:04.776391 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:04.776215    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj_odh-ai-gateway-infra(8027f464-332c-430c-976d-22327de2da7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj" podUID="8027f464-332c-430c-976d-22327de2da7a"
Nov 03 20:00:04.777841 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.777809    5659 generic.go:356] "Generic (PLEG): container finished" podID="3f4bde4d-2410-4104-abaf-edc03cf5e895" containerID="d0492b00c6052cc0636c9a5c087a81d389e9b14359e72138fac2554f9853ebd9" exitCode=60
Nov 03 20:00:04.778085 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.777858    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl" event={"ID":"3f4bde4d-2410-4104-abaf-edc03cf5e895","Type":"ContainerDied","Data":"d0492b00c6052cc0636c9a5c087a81d389e9b14359e72138fac2554f9853ebd9"}
Nov 03 20:00:04.778253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.778238    5659 scope.go:122] "RemoveContainer" containerID="d0492b00c6052cc0636c9a5c087a81d389e9b14359e72138fac2554f9853ebd9"
Nov 03 20:00:04.778712 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:04.778688    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl_odh-ai-gateway-infra(3f4bde4d-2410-4104-abaf-edc03cf5e895)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl" podUID="3f4bde4d-2410-4104-abaf-edc03cf5e895"
Nov 03 20:00:04.780736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.780713    5659 generic.go:356] "Generic (PLEG): container finished" podID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" containerID="892bc066ff365f7d03884d112171321aa29d16ffe1508bde31d995da3d285941" exitCode=60
Nov 03 20:00:04.780880 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.780864    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr" event={"ID":"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c","Type":"ContainerDied","Data":"892bc066ff365f7d03884d112171321aa29d16ffe1508bde31d995da3d285941"}
Nov 03 20:00:04.781267 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.781249    5659 scope.go:122] "RemoveContainer" containerID="892bc066ff365f7d03884d112171321aa29d16ffe1508bde31d995da3d285941"
Nov 03 20:00:04.781660 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:04.781637    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr_odh-ai-gateway-infra(490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr" podUID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c"
Nov 03 20:00:04.786684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.786657    5659 generic.go:356] "Generic (PLEG): container finished" podID="33f77ab6-e954-4124-bd4f-ea0891241124" containerID="b83e152f83f7361f8d73078580580114af33e664c6f602e4b95bb2f9ba943c2a" exitCode=60
Nov 03 20:00:04.786861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.786838    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx" event={"ID":"33f77ab6-e954-4124-bd4f-ea0891241124","Type":"ContainerDied","Data":"b83e152f83f7361f8d73078580580114af33e664c6f602e4b95bb2f9ba943c2a"}
Nov 03 20:00:04.788188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.788167    5659 scope.go:122] "RemoveContainer" containerID="b83e152f83f7361f8d73078580580114af33e664c6f602e4b95bb2f9ba943c2a"
Nov 03 20:00:04.788586 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:04.788562    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx_odh-ai-gateway-infra(33f77ab6-e954-4124-bd4f-ea0891241124)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx" podUID="33f77ab6-e954-4124-bd4f-ea0891241124"
Nov 03 20:00:04.790239 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.790219    5659 scope.go:122] "RemoveContainer" containerID="3e727b9fc22e9724c6b27d505ae1e3bf316c35021c3677691ddd0cfcc24abc43"
Nov 03 20:00:04.804772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.804696    5659 scope.go:122] "RemoveContainer" containerID="93bd37242c2351475639f5aac98be75387f56a358f78caa5692e7f2095bc1dbd"
Nov 03 20:00:04.819885 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.819860    5659 scope.go:122] "RemoveContainer" containerID="f9a4dfadefdfcb0b1847dbfc3ac78629077deb535d1398fd1d76167ef051efc2"
Nov 03 20:00:04.858921 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:04.858334    5659 scope.go:122] "RemoveContainer" containerID="702965e2ecf9eb41987b860c5168f64d2b651bebb639a627c71293bc3f094c36"
Nov 03 20:00:05.448542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:05.442972    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:05.796170 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:05.796137    5659 scope.go:122] "RemoveContainer" containerID="d0492b00c6052cc0636c9a5c087a81d389e9b14359e72138fac2554f9853ebd9"
Nov 03 20:00:05.796900 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:05.796477    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl_odh-ai-gateway-infra(3f4bde4d-2410-4104-abaf-edc03cf5e895)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl" podUID="3f4bde4d-2410-4104-abaf-edc03cf5e895"
Nov 03 20:00:05.798495 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:05.798460    5659 scope.go:122] "RemoveContainer" containerID="892bc066ff365f7d03884d112171321aa29d16ffe1508bde31d995da3d285941"
Nov 03 20:00:05.798812 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:05.798787    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr_odh-ai-gateway-infra(490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr" podUID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c"
Nov 03 20:00:05.800371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:05.800349    5659 scope.go:122] "RemoveContainer" containerID="b83e152f83f7361f8d73078580580114af33e664c6f602e4b95bb2f9ba943c2a"
Nov 03 20:00:05.800694 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:05.800668    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx_odh-ai-gateway-infra(33f77ab6-e954-4124-bd4f-ea0891241124)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx" podUID="33f77ab6-e954-4124-bd4f-ea0891241124"
Nov 03 20:00:05.802208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:05.802189    5659 scope.go:122] "RemoveContainer" containerID="9e3d884bdf9a200ce49271427c0cf90ec694d00053d1a07f79822e31a8eb39a8"
Nov 03 20:00:05.802437 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:05.802414    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895600-tpg7b_odh-ai-gateway-infra(64d32f4c-445e-4aa4-8ecb-bacb6903c699)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b" podUID="64d32f4c-445e-4aa4-8ecb-bacb6903c699"
Nov 03 20:00:05.804104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:05.804086    5659 scope.go:122] "RemoveContainer" containerID="cda7952bf5321b815035c52c83971caab8d48697b9c4c1fa4a2b9d8add3cfe4f"
Nov 03 20:00:05.804334 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:05.804313    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj_odh-ai-gateway-infra(8027f464-332c-430c-976d-22327de2da7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj" podUID="8027f464-332c-430c-976d-22327de2da7a"
Nov 03 20:00:06.841365 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:06.841328    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:06.993503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:06.992955    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:07.163886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:07.163775    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:07.943116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:07.941725    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:08.719588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:08.719311    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:08.879818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:08.877174    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:09.235195 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:09.235165    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:09.624601 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:09.624475    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:10.642390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:10.642363    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:11.436143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:11.436106    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:11.739682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:11.739640    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:12.113603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:12.113511    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:13.041815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:13.041767    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:13.923724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:13.923686    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"]
Nov 03 20:00:13.959785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:13.959752    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:14.095617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.095514    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:14.247156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.247129    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:14.299857 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.299829    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"
Nov 03 20:00:14.483555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.482005    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4bde4d-2410-4104-abaf-edc03cf5e895-kube-api-access-hz86s\" (UniqueName: \"kubernetes.io/projected/3f4bde4d-2410-4104-abaf-edc03cf5e895-kube-api-access-hz86s\") pod \"3f4bde4d-2410-4104-abaf-edc03cf5e895\" (UID: \"3f4bde4d-2410-4104-abaf-edc03cf5e895\") "
Nov 03 20:00:14.483555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.482119    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4bde4d-2410-4104-abaf-edc03cf5e895-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4bde4d-2410-4104-abaf-edc03cf5e895-maas-api-service-ca\") pod \"3f4bde4d-2410-4104-abaf-edc03cf5e895\" (UID: \"3f4bde4d-2410-4104-abaf-edc03cf5e895\") "
Nov 03 20:00:14.483555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.483369    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4bde4d-2410-4104-abaf-edc03cf5e895-maas-api-service-ca" pod "3f4bde4d-2410-4104-abaf-edc03cf5e895" (UID: "3f4bde4d-2410-4104-abaf-edc03cf5e895"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:14.503763 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.503724    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4bde4d-2410-4104-abaf-edc03cf5e895-kube-api-access-hz86s" pod "3f4bde4d-2410-4104-abaf-edc03cf5e895" (UID: "3f4bde4d-2410-4104-abaf-edc03cf5e895"). InnerVolumeSpecName "kube-api-access-hz86s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:14.584244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.584206    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz86s\" (UniqueName: \"kubernetes.io/projected/3f4bde4d-2410-4104-abaf-edc03cf5e895-kube-api-access-hz86s\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:14.584244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.584244    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4bde4d-2410-4104-abaf-edc03cf5e895-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:14.739162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.739038    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:14.957377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.955686    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl" event={"ID":"3f4bde4d-2410-4104-abaf-edc03cf5e895","Type":"ContainerDied","Data":"c0b4e8a8781ff7cab3b07e527575c4f783aa770a8249e7b55662d13865d8cc78"}
Nov 03 20:00:14.957377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.955766    5659 scope.go:122] "RemoveContainer" containerID="d0492b00c6052cc0636c9a5c087a81d389e9b14359e72138fac2554f9853ebd9"
Nov 03 20:00:14.957377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.955920    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"
Nov 03 20:00:14.998056 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:14.997967    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"]
Nov 03 20:00:15.003755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:15.003702    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-277dc9-29895600-wsdrl"]
Nov 03 20:00:15.150725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:15.150560    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:15.226626 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:15.226592    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:00:15.430964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:15.430855    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:15.767612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:15.767101    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:15.961408 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:15.961377    5659 scope.go:122] "RemoveContainer" containerID="b83e152f83f7361f8d73078580580114af33e664c6f602e4b95bb2f9ba943c2a"
Nov 03 20:00:15.973447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:15.973415    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4bde4d-2410-4104-abaf-edc03cf5e895" path="/var/lib/kubelet/pods/3f4bde4d-2410-4104-abaf-edc03cf5e895/volumes"
Nov 03 20:00:16.976505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:16.973783    5659 scope.go:122] "RemoveContainer" containerID="cda7952bf5321b815035c52c83971caab8d48697b9c4c1fa4a2b9d8add3cfe4f"
Nov 03 20:00:16.976505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:16.974152    5659 scope.go:122] "RemoveContainer" containerID="9e3d884bdf9a200ce49271427c0cf90ec694d00053d1a07f79822e31a8eb39a8"
Nov 03 20:00:16.976505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:16.974796    5659 scope.go:122] "RemoveContainer" containerID="892bc066ff365f7d03884d112171321aa29d16ffe1508bde31d995da3d285941"
Nov 03 20:00:17.011058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:17.011030    5659 generic.go:356] "Generic (PLEG): container finished" podID="33f77ab6-e954-4124-bd4f-ea0891241124" containerID="045db70de3ce279f4563b74aae52368ddda75ee2fc0f5b243b6451547ee1b357" exitCode=60
Nov 03 20:00:17.011384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:17.011348    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx" event={"ID":"33f77ab6-e954-4124-bd4f-ea0891241124","Type":"ContainerDied","Data":"045db70de3ce279f4563b74aae52368ddda75ee2fc0f5b243b6451547ee1b357"}
Nov 03 20:00:17.011608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:17.011571    5659 scope.go:122] "RemoveContainer" containerID="b83e152f83f7361f8d73078580580114af33e664c6f602e4b95bb2f9ba943c2a"
Nov 03 20:00:17.054934 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:17.054901    5659 scope.go:122] "RemoveContainer" containerID="045db70de3ce279f4563b74aae52368ddda75ee2fc0f5b243b6451547ee1b357"
Nov 03 20:00:17.055614 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:17.055573    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx_odh-ai-gateway-infra(33f77ab6-e954-4124-bd4f-ea0891241124)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx" podUID="33f77ab6-e954-4124-bd4f-ea0891241124"
Nov 03 20:00:18.021523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.021471    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr" event={"ID":"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c","Type":"ContainerStarted","Data":"12451134d0a35bb97c49d2d7f04a76b79122e836d0893eac8ed7bc7b90d661d9"}
Nov 03 20:00:18.024846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.024822    5659 generic.go:356] "Generic (PLEG): container finished" podID="64d32f4c-445e-4aa4-8ecb-bacb6903c699" containerID="44327384d9f7a8d1f67076ba54679d493c0fe860b995158dbf943d3d71150b75" exitCode=60
Nov 03 20:00:18.024978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.024897    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b" event={"ID":"64d32f4c-445e-4aa4-8ecb-bacb6903c699","Type":"ContainerDied","Data":"44327384d9f7a8d1f67076ba54679d493c0fe860b995158dbf943d3d71150b75"}
Nov 03 20:00:18.024978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.024954    5659 scope.go:122] "RemoveContainer" containerID="9e3d884bdf9a200ce49271427c0cf90ec694d00053d1a07f79822e31a8eb39a8"
Nov 03 20:00:18.025361 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.025330    5659 scope.go:122] "RemoveContainer" containerID="44327384d9f7a8d1f67076ba54679d493c0fe860b995158dbf943d3d71150b75"
Nov 03 20:00:18.025699 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:18.025680    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895600-tpg7b_odh-ai-gateway-infra(64d32f4c-445e-4aa4-8ecb-bacb6903c699)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b" podUID="64d32f4c-445e-4aa4-8ecb-bacb6903c699"
Nov 03 20:00:18.026688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.026666    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj" event={"ID":"8027f464-332c-430c-976d-22327de2da7a","Type":"ContainerStarted","Data":"d97e63afc9f3c495f51c4bc3a0ae44b65c652451415f3006ab7dd84c4b49de02"}
Nov 03 20:00:18.051354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.051325    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"]
Nov 03 20:00:18.235471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.235442    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"
Nov 03 20:00:18.261980 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.261920    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f77ab6-e954-4124-bd4f-ea0891241124-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f77ab6-e954-4124-bd4f-ea0891241124-maas-api-service-ca\") pod \"33f77ab6-e954-4124-bd4f-ea0891241124\" (UID: \"33f77ab6-e954-4124-bd4f-ea0891241124\") "
Nov 03 20:00:18.262145 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.262035    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f77ab6-e954-4124-bd4f-ea0891241124-kube-api-access-wn6p2\" (UniqueName: \"kubernetes.io/projected/33f77ab6-e954-4124-bd4f-ea0891241124-kube-api-access-wn6p2\") pod \"33f77ab6-e954-4124-bd4f-ea0891241124\" (UID: \"33f77ab6-e954-4124-bd4f-ea0891241124\") "
Nov 03 20:00:18.262634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.262603    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f77ab6-e954-4124-bd4f-ea0891241124-maas-api-service-ca" pod "33f77ab6-e954-4124-bd4f-ea0891241124" (UID: "33f77ab6-e954-4124-bd4f-ea0891241124"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:18.262864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.262843    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f77ab6-e954-4124-bd4f-ea0891241124-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:18.265157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.265129    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f77ab6-e954-4124-bd4f-ea0891241124-kube-api-access-wn6p2" pod "33f77ab6-e954-4124-bd4f-ea0891241124" (UID: "33f77ab6-e954-4124-bd4f-ea0891241124"). InnerVolumeSpecName "kube-api-access-wn6p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:18.366165 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:18.366125    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn6p2\" (UniqueName: \"kubernetes.io/projected/33f77ab6-e954-4124-bd4f-ea0891241124-kube-api-access-wn6p2\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.056977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.056935    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"]
Nov 03 20:00:19.076046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.075956    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"]
Nov 03 20:00:19.101512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.095983    5659 generic.go:356] "Generic (PLEG): container finished" podID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" containerID="12451134d0a35bb97c49d2d7f04a76b79122e836d0893eac8ed7bc7b90d661d9" exitCode=60
Nov 03 20:00:19.101512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.096160    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr" event={"ID":"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c","Type":"ContainerDied","Data":"12451134d0a35bb97c49d2d7f04a76b79122e836d0893eac8ed7bc7b90d661d9"}
Nov 03 20:00:19.101512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.096217    5659 scope.go:122] "RemoveContainer" containerID="892bc066ff365f7d03884d112171321aa29d16ffe1508bde31d995da3d285941"
Nov 03 20:00:19.101512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.100231    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"
Nov 03 20:00:19.102440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.102417    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx" event={"ID":"33f77ab6-e954-4124-bd4f-ea0891241124","Type":"ContainerDied","Data":"5a108b0086993addec92c4f22c1092ec090a388c9aa53346db2dd1560fed65ee"}
Nov 03 20:00:19.102668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.102650    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"]
Nov 03 20:00:19.125507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.125407    5659 generic.go:356] "Generic (PLEG): container finished" podID="8027f464-332c-430c-976d-22327de2da7a" containerID="d97e63afc9f3c495f51c4bc3a0ae44b65c652451415f3006ab7dd84c4b49de02" exitCode=60
Nov 03 20:00:19.125507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.125504    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj" event={"ID":"8027f464-332c-430c-976d-22327de2da7a","Type":"ContainerDied","Data":"d97e63afc9f3c495f51c4bc3a0ae44b65c652451415f3006ab7dd84c4b49de02"}
Nov 03 20:00:19.140807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.140781    5659 scope.go:122] "RemoveContainer" containerID="045db70de3ce279f4563b74aae52368ddda75ee2fc0f5b243b6451547ee1b357"
Nov 03 20:00:19.196206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.195863    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"]
Nov 03 20:00:19.201667 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.201639    5659 scope.go:122] "RemoveContainer" containerID="cda7952bf5321b815035c52c83971caab8d48697b9c4c1fa4a2b9d8add3cfe4f"
Nov 03 20:00:19.209786 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.204132    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:19.212695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.212502    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3faa07-29895600-9b7zx"]
Nov 03 20:00:19.348748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.348709    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:19.458186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.457870    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"
Nov 03 20:00:19.482734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.482668    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:19.538680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.538652    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d32f4c-445e-4aa4-8ecb-bacb6903c699-kube-api-access-5qsg2\" (UniqueName: \"kubernetes.io/projected/64d32f4c-445e-4aa4-8ecb-bacb6903c699-kube-api-access-5qsg2\") pod \"64d32f4c-445e-4aa4-8ecb-bacb6903c699\" (UID: \"64d32f4c-445e-4aa4-8ecb-bacb6903c699\") "
Nov 03 20:00:19.538811 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.538724    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d32f4c-445e-4aa4-8ecb-bacb6903c699-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d32f4c-445e-4aa4-8ecb-bacb6903c699-maas-api-service-ca\") pod \"64d32f4c-445e-4aa4-8ecb-bacb6903c699\" (UID: \"64d32f4c-445e-4aa4-8ecb-bacb6903c699\") "
Nov 03 20:00:19.539805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.539763    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d32f4c-445e-4aa4-8ecb-bacb6903c699-maas-api-service-ca" pod "64d32f4c-445e-4aa4-8ecb-bacb6903c699" (UID: "64d32f4c-445e-4aa4-8ecb-bacb6903c699"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:19.540249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.540224    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d32f4c-445e-4aa4-8ecb-bacb6903c699-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.542142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.542117    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d32f4c-445e-4aa4-8ecb-bacb6903c699-kube-api-access-5qsg2" pod "64d32f4c-445e-4aa4-8ecb-bacb6903c699" (UID: "64d32f4c-445e-4aa4-8ecb-bacb6903c699"). InnerVolumeSpecName "kube-api-access-5qsg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:19.569765 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.569737    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"
Nov 03 20:00:19.571792 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.571775    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"
Nov 03 20:00:19.625001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.624913    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:19.641031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.640995    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8027f464-332c-430c-976d-22327de2da7a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8027f464-332c-430c-976d-22327de2da7a-maas-api-service-ca\") pod \"8027f464-332c-430c-976d-22327de2da7a\" (UID: \"8027f464-332c-430c-976d-22327de2da7a\") "
Nov 03 20:00:19.641196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.641085    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-kube-api-access-56d52\" (UniqueName: \"kubernetes.io/projected/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-kube-api-access-56d52\") pod \"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c\" (UID: \"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c\") "
Nov 03 20:00:19.641196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.641133    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-maas-api-service-ca\") pod \"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c\" (UID: \"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c\") "
Nov 03 20:00:19.641410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.641382    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8027f464-332c-430c-976d-22327de2da7a-maas-api-service-ca" pod "8027f464-332c-430c-976d-22327de2da7a" (UID: "8027f464-332c-430c-976d-22327de2da7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:19.641661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.641396    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8027f464-332c-430c-976d-22327de2da7a-kube-api-access-9lq2d\" (UniqueName: \"kubernetes.io/projected/8027f464-332c-430c-976d-22327de2da7a-kube-api-access-9lq2d\") pod \"8027f464-332c-430c-976d-22327de2da7a\" (UID: \"8027f464-332c-430c-976d-22327de2da7a\") "
Nov 03 20:00:19.642210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.641717    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-maas-api-service-ca" pod "490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" (UID: "490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:19.642608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.642580    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qsg2\" (UniqueName: \"kubernetes.io/projected/64d32f4c-445e-4aa4-8ecb-bacb6903c699-kube-api-access-5qsg2\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.642722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.642610    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8027f464-332c-430c-976d-22327de2da7a-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.642722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.642625    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.643780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.643756    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8027f464-332c-430c-976d-22327de2da7a-kube-api-access-9lq2d" pod "8027f464-332c-430c-976d-22327de2da7a" (UID: "8027f464-332c-430c-976d-22327de2da7a"). InnerVolumeSpecName "kube-api-access-9lq2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:19.644787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.644766    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-kube-api-access-56d52" pod "490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" (UID: "490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c"). InnerVolumeSpecName "kube-api-access-56d52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:19.798727 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.782611    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lq2d\" (UniqueName: \"kubernetes.io/projected/8027f464-332c-430c-976d-22327de2da7a-kube-api-access-9lq2d\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.798727 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.782650    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56d52\" (UniqueName: \"kubernetes.io/projected/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c-kube-api-access-56d52\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.970693 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:19.970656    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f77ab6-e954-4124-bd4f-ea0891241124" path="/var/lib/kubelet/pods/33f77ab6-e954-4124-bd4f-ea0891241124/volumes"
Nov 03 20:00:20.133270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.133241    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"
Nov 03 20:00:20.133794 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.133293    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr" event={"ID":"490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c","Type":"ContainerDied","Data":"39c78a3a5c668850a25e9f185042bd12178227520ef02f6861c93072ffd96ffe"}
Nov 03 20:00:20.133794 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.133400    5659 scope.go:122] "RemoveContainer" containerID="12451134d0a35bb97c49d2d7f04a76b79122e836d0893eac8ed7bc7b90d661d9"
Nov 03 20:00:20.136188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.136166    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"
Nov 03 20:00:20.136293 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.136197    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b" event={"ID":"64d32f4c-445e-4aa4-8ecb-bacb6903c699","Type":"ContainerDied","Data":"ed759b344ad373d563effa1a1cd5f9d2dbbb286be7b3ab97ef5d7c3dd5b0a524"}
Nov 03 20:00:20.138678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.138652    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj" event={"ID":"8027f464-332c-430c-976d-22327de2da7a","Type":"ContainerDied","Data":"e78761f8d06b505f043b217c60514f646c58f811ee69a43912978a562e5f8cc9"}
Nov 03 20:00:20.138791 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.138745    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"
Nov 03 20:00:20.146427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.146409    5659 scope.go:122] "RemoveContainer" containerID="44327384d9f7a8d1f67076ba54679d493c0fe860b995158dbf943d3d71150b75"
Nov 03 20:00:20.155283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.155267    5659 scope.go:122] "RemoveContainer" containerID="d97e63afc9f3c495f51c4bc3a0ae44b65c652451415f3006ab7dd84c4b49de02"
Nov 03 20:00:20.182962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.182939    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"]
Nov 03 20:00:20.246731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.246646    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3d1db1-29895600-v4qvr"]
Nov 03 20:00:20.295441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.295407    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"]
Nov 03 20:00:20.307076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.307038    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-tpg7b"]
Nov 03 20:00:20.321635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.321605    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"]
Nov 03 20:00:20.330095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.330063    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5cf30a-29895600-9wmzj"]
Nov 03 20:00:20.756581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.756548    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:20.892832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:20.892800    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:21.826822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:21.826789    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:21.970104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:21.970073    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" path="/var/lib/kubelet/pods/490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c/volumes"
Nov 03 20:00:21.970553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:21.970539    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d32f4c-445e-4aa4-8ecb-bacb6903c699" path="/var/lib/kubelet/pods/64d32f4c-445e-4aa4-8ecb-bacb6903c699/volumes"
Nov 03 20:00:21.970994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:21.970978    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8027f464-332c-430c-976d-22327de2da7a" path="/var/lib/kubelet/pods/8027f464-332c-430c-976d-22327de2da7a/volumes"
Nov 03 20:00:21.986772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:21.986748    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:22.237510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.236982    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:22.288103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.284896    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"]
Nov 03 20:00:22.288103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.285174    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh" podUID="56c05f1f-b29e-4346-b5e3-69381c6b626f" containerName="maas-api" containerID="cri-o://32bc1089ca279406104de34cf1f0cda5cc9d2dd93a0e218efb92b8b59184568e" gracePeriod=30
Nov 03 20:00:22.399256 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.399157    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:22.609532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.597981    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:22.683505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.672698    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 20:00:22.762579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.762304    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:22.762579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.762356    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-maas-api-tls\") pod \"56c05f1f-b29e-4346-b5e3-69381c6b626f\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") "
Nov 03 20:00:22.762579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.762417    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c05f1f-b29e-4346-b5e3-69381c6b626f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c05f1f-b29e-4346-b5e3-69381c6b626f-openshift-service-ca\") pod \"56c05f1f-b29e-4346-b5e3-69381c6b626f\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") "
Nov 03 20:00:22.762579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.762533    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-metrics-certs\") pod \"56c05f1f-b29e-4346-b5e3-69381c6b626f\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") "
Nov 03 20:00:22.762928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.762606    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c05f1f-b29e-4346-b5e3-69381c6b626f-kube-api-access-hd5gs\" (UniqueName: \"kubernetes.io/projected/56c05f1f-b29e-4346-b5e3-69381c6b626f-kube-api-access-hd5gs\") pod \"56c05f1f-b29e-4346-b5e3-69381c6b626f\" (UID: \"56c05f1f-b29e-4346-b5e3-69381c6b626f\") "
Nov 03 20:00:22.768876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.765923    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c05f1f-b29e-4346-b5e3-69381c6b626f-openshift-service-ca" pod "56c05f1f-b29e-4346-b5e3-69381c6b626f" (UID: "56c05f1f-b29e-4346-b5e3-69381c6b626f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:22.768876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.767164    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c05f1f-b29e-4346-b5e3-69381c6b626f-kube-api-access-hd5gs" pod "56c05f1f-b29e-4346-b5e3-69381c6b626f" (UID: "56c05f1f-b29e-4346-b5e3-69381c6b626f"). InnerVolumeSpecName "kube-api-access-hd5gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:22.799541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.788232    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-metrics-certs" pod "56c05f1f-b29e-4346-b5e3-69381c6b626f" (UID: "56c05f1f-b29e-4346-b5e3-69381c6b626f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:00:22.799541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.788343    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-maas-api-tls" pod "56c05f1f-b29e-4346-b5e3-69381c6b626f" (UID: "56c05f1f-b29e-4346-b5e3-69381c6b626f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:00:22.865872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.865836    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:22.865872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.865870    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c05f1f-b29e-4346-b5e3-69381c6b626f-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:22.866474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.865887    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56c05f1f-b29e-4346-b5e3-69381c6b626f-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:22.866474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:22.865899    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd5gs\" (UniqueName: \"kubernetes.io/projected/56c05f1f-b29e-4346-b5e3-69381c6b626f-kube-api-access-hd5gs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:23.080744 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:23.080638    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:23.165699 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:23.165660    5659 generic.go:356] "Generic (PLEG): container finished" podID="56c05f1f-b29e-4346-b5e3-69381c6b626f" containerID="32bc1089ca279406104de34cf1f0cda5cc9d2dd93a0e218efb92b8b59184568e" exitCode=0
Nov 03 20:00:23.165894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:23.165716    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"
Nov 03 20:00:23.165894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:23.165754    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh" event={"ID":"56c05f1f-b29e-4346-b5e3-69381c6b626f","Type":"ContainerDied","Data":"32bc1089ca279406104de34cf1f0cda5cc9d2dd93a0e218efb92b8b59184568e"}
Nov 03 20:00:23.165894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:23.165836    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh" event={"ID":"56c05f1f-b29e-4346-b5e3-69381c6b626f","Type":"ContainerDied","Data":"bca232cca06d1cc2fa82fd3249bb487731cfc54a9efd25d8f96aaa2f34740255"}
Nov 03 20:00:23.165894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:23.165868    5659 scope.go:122] "RemoveContainer" containerID="32bc1089ca279406104de34cf1f0cda5cc9d2dd93a0e218efb92b8b59184568e"
Nov 03 20:00:23.184115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:23.184094    5659 scope.go:122] "RemoveContainer" containerID="32bc1089ca279406104de34cf1f0cda5cc9d2dd93a0e218efb92b8b59184568e"
Nov 03 20:00:23.184407 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:23.184385    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32bc1089ca279406104de34cf1f0cda5cc9d2dd93a0e218efb92b8b59184568e\": container with ID starting with 32bc1089ca279406104de34cf1f0cda5cc9d2dd93a0e218efb92b8b59184568e not found: ID does not exist" containerID="32bc1089ca279406104de34cf1f0cda5cc9d2dd93a0e218efb92b8b59184568e"
Nov 03 20:00:23.184506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:23.184418    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32bc1089ca279406104de34cf1f0cda5cc9d2dd93a0e218efb92b8b59184568e"} err="failed to get container status \"32bc1089ca279406104de34cf1f0cda5cc9d2dd93a0e218efb92b8b59184568e\": rpc error: code = NotFound desc = could not find container \"32bc1089ca279406104de34cf1f0cda5cc9d2dd93a0e218efb92b8b59184568e\": container with ID starting with 32bc1089ca279406104de34cf1f0cda5cc9d2dd93a0e218efb92b8b59184568e not found: ID does not exist"
Nov 03 20:00:23.195591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:23.195568    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"]
Nov 03 20:00:23.200782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:23.200759    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3faa07-59646f59df-dpwhh"]
Nov 03 20:00:23.971301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:23.971268    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c05f1f-b29e-4346-b5e3-69381c6b626f" path="/var/lib/kubelet/pods/56c05f1f-b29e-4346-b5e3-69381c6b626f/volumes"
Nov 03 20:00:24.123835 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:24.123804    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:24.879563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:24.879525    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:25.685530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:25.685474    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:26.622249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:26.622217    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:27.199715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:27.199680    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:27.326178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:27.326140    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:27.468275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:27.468241    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:27.729600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:27.729513    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:28.140978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:28.140895    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:28.599583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:28.599549    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:28.912174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:28.912147    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:29.432033 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:29.432001    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:29.569968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:29.569931    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:30.086071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.086038    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"]
Nov 03 20:00:30.086827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.086270    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f" podUID="82dd5715-f5dc-492e-8764-0c8fd945fbe7" containerName="maas-api" containerID="cri-o://723beb9bd24d6e361d98a72beef0950cc704b5d4bce58b77c6d39c4bbb248a76" gracePeriod=30
Nov 03 20:00:30.168993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.168962    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:30.250903 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.250873    5659 generic.go:356] "Generic (PLEG): container finished" podID="82dd5715-f5dc-492e-8764-0c8fd945fbe7" containerID="723beb9bd24d6e361d98a72beef0950cc704b5d4bce58b77c6d39c4bbb248a76" exitCode=0
Nov 03 20:00:30.251255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.251233    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f" event={"ID":"82dd5715-f5dc-492e-8764-0c8fd945fbe7","Type":"ContainerDied","Data":"723beb9bd24d6e361d98a72beef0950cc704b5d4bce58b77c6d39c4bbb248a76"}
Nov 03 20:00:30.295810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.295781    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:30.650670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.650637    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:30.674273 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.674245    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 20:00:30.730510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.730461    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-metrics-certs\") pod \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") "
Nov 03 20:00:30.730667 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.730520    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82dd5715-f5dc-492e-8764-0c8fd945fbe7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82dd5715-f5dc-492e-8764-0c8fd945fbe7-openshift-service-ca\") pod \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") "
Nov 03 20:00:30.730667 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.730612    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-maas-api-tls\") pod \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") "
Nov 03 20:00:30.730779 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.730725    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82dd5715-f5dc-492e-8764-0c8fd945fbe7-kube-api-access-svd2g\" (UniqueName: \"kubernetes.io/projected/82dd5715-f5dc-492e-8764-0c8fd945fbe7-kube-api-access-svd2g\") pod \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\" (UID: \"82dd5715-f5dc-492e-8764-0c8fd945fbe7\") "
Nov 03 20:00:30.730974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.730947    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82dd5715-f5dc-492e-8764-0c8fd945fbe7-openshift-service-ca" pod "82dd5715-f5dc-492e-8764-0c8fd945fbe7" (UID: "82dd5715-f5dc-492e-8764-0c8fd945fbe7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:30.731351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.731324    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82dd5715-f5dc-492e-8764-0c8fd945fbe7-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:30.733173 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.733141    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-maas-api-tls" pod "82dd5715-f5dc-492e-8764-0c8fd945fbe7" (UID: "82dd5715-f5dc-492e-8764-0c8fd945fbe7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:00:30.733505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.733469    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-metrics-certs" pod "82dd5715-f5dc-492e-8764-0c8fd945fbe7" (UID: "82dd5715-f5dc-492e-8764-0c8fd945fbe7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:00:30.733777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.733756    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dd5715-f5dc-492e-8764-0c8fd945fbe7-kube-api-access-svd2g" pod "82dd5715-f5dc-492e-8764-0c8fd945fbe7" (UID: "82dd5715-f5dc-492e-8764-0c8fd945fbe7"). InnerVolumeSpecName "kube-api-access-svd2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:30.832910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.832831    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:30.832910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.832865    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svd2g\" (UniqueName: \"kubernetes.io/projected/82dd5715-f5dc-492e-8764-0c8fd945fbe7-kube-api-access-svd2g\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:30.832910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:30.832878    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82dd5715-f5dc-492e-8764-0c8fd945fbe7-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:31.276761 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:31.276720    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f" event={"ID":"82dd5715-f5dc-492e-8764-0c8fd945fbe7","Type":"ContainerDied","Data":"5dfe04f856eacaad3a60df8a77d0cb9d8b32b8d12e3ac8dca950d15fe3703bd2"}
Nov 03 20:00:31.277770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:31.277750    5659 scope.go:122] "RemoveContainer" containerID="723beb9bd24d6e361d98a72beef0950cc704b5d4bce58b77c6d39c4bbb248a76"
Nov 03 20:00:31.278012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:31.277516    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"
Nov 03 20:00:31.385559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:31.379938    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"]
Nov 03 20:00:31.391362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:31.391310    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f"]
Nov 03 20:00:31.497942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:31.497901    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:31.973995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:31.973955    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82dd5715-f5dc-492e-8764-0c8fd945fbe7" path="/var/lib/kubelet/pods/82dd5715-f5dc-492e-8764-0c8fd945fbe7/volumes"
Nov 03 20:00:32.141812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:32.141777    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:32.458567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:32.458403    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:32.604079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:32.604045    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:32.915239 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:32.915147    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:33.630627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:33.630582    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-389e6d-5fc5889fd8-4dc7f" podUID="82dd5715-f5dc-492e-8764-0c8fd945fbe7" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.78:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:00:33.865763 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:33.865726    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:35.010163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:35.010134    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:35.861017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:35.860957    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:36.035825 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:36.035791    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:36.183349 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:36.183256    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:36.445131 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:36.445044    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:36.570898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:36.570866    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:36.982734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:36.982691    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:37.325903 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:37.325820    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:37.468972 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:37.468939    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:38.838794 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:38.838762    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:39.365706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:39.365670    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:39.821672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:39.821638    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:40.491507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:40.491426    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:41.360601 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:41.360554    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:42.255893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:42.255854    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:42.800351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:42.800304    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:42.940982 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:42.940944    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:43.084529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:43.084405    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:43.214112 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:43.214077    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:43.377095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:43.377010    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:43.734670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:43.734622    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:43.873020 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:43.872985    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:44.405745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:44.405701    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:44.698784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:44.698704    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:45.222961 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:00:45.222909    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:00:45.548117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:45.548039    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:46.393806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:46.393773    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:47.155590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:47.155539    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:48.036753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:48.036720    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:48.753627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:48.753593    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:48.904360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:48.904324    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:49.052334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:49.052229    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:49.264940 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:49.264907    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:49.848220 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:49.848185    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:49.987410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:49.987371    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:50.205708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:50.205680    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:50.656348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:50.656312    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:51.447501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:51.447444    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:52.251269 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:52.251235    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:53.147054 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:53.147018    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:54.178669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:54.178580    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:54.850143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:54.850104    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:55.629934 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:55.629904    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:56.336348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:56.336308    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:56.574905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:56.574869    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:56.708067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:56.708041    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:57.143398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:57.143299    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:57.272073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:57.272034    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:57.482984 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:57.482952    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"]
Nov 03 20:00:57.483532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:57.483461    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy" containerID="cri-o://ba87f529ff35b90e8c98f1fe22095b1a8c6b3a55b720d255e56129b7c0f2fd10" gracePeriod=30
Nov 03 20:00:57.537474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:57.537439    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:58.157251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:58.157217    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:59.114388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:59.114345    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:59.740555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:59.740520    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:59.991897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:00:59.991781    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:01.246509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:01.233125    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:01.675562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:01.672594    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:01.810521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:01.809430    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:02.232502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.202332    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:02.257594 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.247325    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:02.558281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.558237    5659 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:01:02.558511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.558343    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 20:01:02.704249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.704215    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 20:01:02.794032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.793945    5659 generic.go:356] "Generic (PLEG): container finished" podID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerID="ba87f529ff35b90e8c98f1fe22095b1a8c6b3a55b720d255e56129b7c0f2fd10" exitCode=0
Nov 03 20:01:02.794032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.794002    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"
Nov 03 20:01:02.794217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.794036    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" event={"ID":"a49ffc14-16a0-495d-954f-a524e0d4a81a","Type":"ContainerDied","Data":"ba87f529ff35b90e8c98f1fe22095b1a8c6b3a55b720d255e56129b7c0f2fd10"}
Nov 03 20:01:02.794217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.794096    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz" event={"ID":"a49ffc14-16a0-495d-954f-a524e0d4a81a","Type":"ContainerDied","Data":"d1f1301b4c2af1319db1626d2ea8a2c77b8ba42793be339cd356982b122ff999"}
Nov 03 20:01:02.794217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.794119    5659 scope.go:122] "RemoveContainer" containerID="ba87f529ff35b90e8c98f1fe22095b1a8c6b3a55b720d255e56129b7c0f2fd10"
Nov 03 20:01:02.804152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.804132    5659 scope.go:122] "RemoveContainer" containerID="4fa08e00988778f86683bc137aa8a81af9a8e5dad6f8cbcac1803a13915ba093"
Nov 03 20:01:02.815524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.815502    5659 scope.go:122] "RemoveContainer" containerID="ba87f529ff35b90e8c98f1fe22095b1a8c6b3a55b720d255e56129b7c0f2fd10"
Nov 03 20:01:02.815841 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:01:02.815816    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba87f529ff35b90e8c98f1fe22095b1a8c6b3a55b720d255e56129b7c0f2fd10\": container with ID starting with ba87f529ff35b90e8c98f1fe22095b1a8c6b3a55b720d255e56129b7c0f2fd10 not found: ID does not exist" containerID="ba87f529ff35b90e8c98f1fe22095b1a8c6b3a55b720d255e56129b7c0f2fd10"
Nov 03 20:01:02.815926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.815847    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba87f529ff35b90e8c98f1fe22095b1a8c6b3a55b720d255e56129b7c0f2fd10"} err="failed to get container status \"ba87f529ff35b90e8c98f1fe22095b1a8c6b3a55b720d255e56129b7c0f2fd10\": rpc error: code = NotFound desc = could not find container \"ba87f529ff35b90e8c98f1fe22095b1a8c6b3a55b720d255e56129b7c0f2fd10\": container with ID starting with ba87f529ff35b90e8c98f1fe22095b1a8c6b3a55b720d255e56129b7c0f2fd10 not found: ID does not exist"
Nov 03 20:01:02.815926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.815866    5659 scope.go:122] "RemoveContainer" containerID="4fa08e00988778f86683bc137aa8a81af9a8e5dad6f8cbcac1803a13915ba093"
Nov 03 20:01:02.816124 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:01:02.816108    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa08e00988778f86683bc137aa8a81af9a8e5dad6f8cbcac1803a13915ba093\": container with ID starting with 4fa08e00988778f86683bc137aa8a81af9a8e5dad6f8cbcac1803a13915ba093 not found: ID does not exist" containerID="4fa08e00988778f86683bc137aa8a81af9a8e5dad6f8cbcac1803a13915ba093"
Nov 03 20:01:02.816174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.816136    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa08e00988778f86683bc137aa8a81af9a8e5dad6f8cbcac1803a13915ba093"} err="failed to get container status \"4fa08e00988778f86683bc137aa8a81af9a8e5dad6f8cbcac1803a13915ba093\": rpc error: code = NotFound desc = could not find container \"4fa08e00988778f86683bc137aa8a81af9a8e5dad6f8cbcac1803a13915ba093\": container with ID starting with 4fa08e00988778f86683bc137aa8a81af9a8e5dad6f8cbcac1803a13915ba093 not found: ID does not exist"
Nov 03 20:01:02.820206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.820186    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-token\" (UniqueName: \"kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-token\") pod \"a49ffc14-16a0-495d-954f-a524e0d4a81a\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") "
Nov 03 20:01:02.820307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.820294    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-credential-socket\") pod \"a49ffc14-16a0-495d-954f-a524e0d4a81a\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") "
Nov 03 20:01:02.820353 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.820330    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-envoy\") pod \"a49ffc14-16a0-495d-954f-a524e0d4a81a\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") "
Nov 03 20:01:02.820407 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.820396    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a49ffc14-16a0-495d-954f-a524e0d4a81a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49ffc14-16a0-495d-954f-a524e0d4a81a-istiod-ca-cert\") pod \"a49ffc14-16a0-495d-954f-a524e0d4a81a\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") "
Nov 03 20:01:02.820456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.820417    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-data\") pod \"a49ffc14-16a0-495d-954f-a524e0d4a81a\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") "
Nov 03 20:01:02.820456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.820436    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-podinfo\") pod \"a49ffc14-16a0-495d-954f-a524e0d4a81a\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") "
Nov 03 20:01:02.820596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.820464    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-certs\") pod \"a49ffc14-16a0-495d-954f-a524e0d4a81a\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") "
Nov 03 20:01:02.820596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.820540    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-kube-api-access-5shrt\" (UniqueName: \"kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-kube-api-access-5shrt\") pod \"a49ffc14-16a0-495d-954f-a524e0d4a81a\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") "
Nov 03 20:01:02.820707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.820594    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-credential-socket" pod "a49ffc14-16a0-495d-954f-a524e0d4a81a" (UID: "a49ffc14-16a0-495d-954f-a524e0d4a81a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:02.820707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.820620    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-socket\") pod \"a49ffc14-16a0-495d-954f-a524e0d4a81a\" (UID: \"a49ffc14-16a0-495d-954f-a524e0d4a81a\") "
Nov 03 20:01:02.820815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.820748    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-data" pod "a49ffc14-16a0-495d-954f-a524e0d4a81a" (UID: "a49ffc14-16a0-495d-954f-a524e0d4a81a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:02.820815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.820768    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-certs" pod "a49ffc14-16a0-495d-954f-a524e0d4a81a" (UID: "a49ffc14-16a0-495d-954f-a524e0d4a81a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:02.820924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.820829    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49ffc14-16a0-495d-954f-a524e0d4a81a-istiod-ca-cert" pod "a49ffc14-16a0-495d-954f-a524e0d4a81a" (UID: "a49ffc14-16a0-495d-954f-a524e0d4a81a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:02.821106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.821081    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-socket" pod "a49ffc14-16a0-495d-954f-a524e0d4a81a" (UID: "a49ffc14-16a0-495d-954f-a524e0d4a81a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:02.821370 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.821346    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:02.821461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.821377    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49ffc14-16a0-495d-954f-a524e0d4a81a-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:02.821461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.821392    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:02.821461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.821407    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:02.821461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.821421    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:02.822696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.822677    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-token" pod "a49ffc14-16a0-495d-954f-a524e0d4a81a" (UID: "a49ffc14-16a0-495d-954f-a524e0d4a81a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:02.822771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.822722    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-kube-api-access-5shrt" pod "a49ffc14-16a0-495d-954f-a524e0d4a81a" (UID: "a49ffc14-16a0-495d-954f-a524e0d4a81a"). InnerVolumeSpecName "kube-api-access-5shrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:02.822771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.822722    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-envoy" pod "a49ffc14-16a0-495d-954f-a524e0d4a81a" (UID: "a49ffc14-16a0-495d-954f-a524e0d4a81a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:02.822771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.822739    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-podinfo" pod "a49ffc14-16a0-495d-954f-a524e0d4a81a" (UID: "a49ffc14-16a0-495d-954f-a524e0d4a81a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:01:02.922410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.922374    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:02.922410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.922410    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:02.922668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.922425    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5shrt\" (UniqueName: \"kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-kube-api-access-5shrt\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:02.922668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.922436    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49ffc14-16a0-495d-954f-a524e0d4a81a-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:02.980292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:02.980253    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:03.124163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:03.124122    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"]
Nov 03 20:01:03.128306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:03.128275    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-mq6jz"]
Nov 03 20:01:03.956444 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:03.956411    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:03.993256 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:03.993207    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" path="/var/lib/kubelet/pods/a49ffc14-16a0-495d-954f-a524e0d4a81a/volumes"
Nov 03 20:01:04.114080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:04.114043    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:04.652080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:04.652050    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:05.453986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:05.453935    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:06.289172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:06.289138    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:06.427738 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:06.427705    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:07.134632 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:07.134578    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:07.800538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:07.800465    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:07.926198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:07.926161    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:08.386809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:08.386774    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:09.128644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:09.128609    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:09.804166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:09.804136    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:10.540750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:10.540715    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:11.356539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:11.356470    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:12.127442 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:12.127397    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:12.525782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:12.525746    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:12.647901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:12.647867    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:12.999993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:12.999956    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:13.147868 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:13.147835    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:13.800900 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:13.800860    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:14.561097 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:14.561063    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:15.217666 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:01:15.217630    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:01:15.337518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:15.337446    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:16.046237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:16.046206    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:16.821922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:16.821878    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:17.746723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:17.746690    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:18.553036 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:18.552975    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:19.326307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:19.326251    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:20.027925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.027894    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:20.593270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.593233    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"]
Nov 03 20:01:20.594029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.593967    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3f4bde4d-2410-4104-abaf-edc03cf5e895" containerName="cleanup"
Nov 03 20:01:20.594029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594022    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4bde4d-2410-4104-abaf-edc03cf5e895" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594038    5659 cpu_manager.go:414] "containerMap: removing container" podUID="56c05f1f-b29e-4346-b5e3-69381c6b626f" containerName="maas-api"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594047    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c05f1f-b29e-4346-b5e3-69381c6b626f" containerName="maas-api"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594064    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3f4bde4d-2410-4104-abaf-edc03cf5e895" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594070    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4bde4d-2410-4104-abaf-edc03cf5e895" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594077    5659 cpu_manager.go:414] "containerMap: removing container" podUID="33f77ab6-e954-4124-bd4f-ea0891241124" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594082    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f77ab6-e954-4124-bd4f-ea0891241124" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594087    5659 cpu_manager.go:414] "containerMap: removing container" podUID="64d32f4c-445e-4aa4-8ecb-bacb6903c699" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594093    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d32f4c-445e-4aa4-8ecb-bacb6903c699" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594100    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8027f464-332c-430c-976d-22327de2da7a" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594105    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8027f464-332c-430c-976d-22327de2da7a" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594114    5659 cpu_manager.go:414] "containerMap: removing container" podUID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594120    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594131    5659 cpu_manager.go:414] "containerMap: removing container" podUID="64d32f4c-445e-4aa4-8ecb-bacb6903c699" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594135    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d32f4c-445e-4aa4-8ecb-bacb6903c699" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594142    5659 cpu_manager.go:414] "containerMap: removing container" podUID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594147    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594160    5659 cpu_manager.go:414] "containerMap: removing container" podUID="82dd5715-f5dc-492e-8764-0c8fd945fbe7" containerName="maas-api"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594166    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82dd5715-f5dc-492e-8764-0c8fd945fbe7" containerName="maas-api"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594173    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594177    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594188    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594195    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594218    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8027f464-332c-430c-976d-22327de2da7a" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594223    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8027f464-332c-430c-976d-22327de2da7a" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594230    5659 cpu_manager.go:414] "containerMap: removing container" podUID="33f77ab6-e954-4124-bd4f-ea0891241124" containerName="cleanup"
Nov 03 20:01:20.594246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594234    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f77ab6-e954-4124-bd4f-ea0891241124" containerName="cleanup"
Nov 03 20:01:20.595156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594361    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d32f4c-445e-4aa4-8ecb-bacb6903c699" containerName="cleanup"
Nov 03 20:01:20.595156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594372    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" containerName="cleanup"
Nov 03 20:01:20.595156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594384    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="82dd5715-f5dc-492e-8764-0c8fd945fbe7" containerName="maas-api"
Nov 03 20:01:20.595156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594392    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy"
Nov 03 20:01:20.595156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594399    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f77ab6-e954-4124-bd4f-ea0891241124" containerName="cleanup"
Nov 03 20:01:20.595156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594408    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" containerName="cleanup"
Nov 03 20:01:20.595156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594418    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8027f464-332c-430c-976d-22327de2da7a" containerName="cleanup"
Nov 03 20:01:20.595156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594427    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8027f464-332c-430c-976d-22327de2da7a" containerName="cleanup"
Nov 03 20:01:20.595156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594438    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4bde4d-2410-4104-abaf-edc03cf5e895" containerName="cleanup"
Nov 03 20:01:20.595156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594445    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c05f1f-b29e-4346-b5e3-69381c6b626f" containerName="maas-api"
Nov 03 20:01:20.595156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594451    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4bde4d-2410-4104-abaf-edc03cf5e895" containerName="cleanup"
Nov 03 20:01:20.595156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594458    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f77ab6-e954-4124-bd4f-ea0891241124" containerName="cleanup"
Nov 03 20:01:20.595156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.594462    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d32f4c-445e-4aa4-8ecb-bacb6903c699" containerName="cleanup"
Nov 03 20:01:20.597855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.597832    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.612982 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.612955    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"]
Nov 03 20:01:20.750700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.750665    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:20.773960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.773917    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.774165 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.773974    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.774165 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.774069    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.774297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.774168    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlzxr\" (UniqueName: \"kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-kube-api-access-qlzxr\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.774297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.774265    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.774297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.774289    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.774451 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.774321    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.774540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.774459    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.774599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.774576    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.877565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.876688    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.877565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.876764    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.877565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.876822    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.877565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.876941    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlzxr\" (UniqueName: \"kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-kube-api-access-qlzxr\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.877565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.876990    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.877565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.877015    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.877565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.877063    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.877565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.877127    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.877565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.877207    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.878141 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.878092    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.880532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.878372    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.881878 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.881848    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.885676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.882284    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.885676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.882471    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.885676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.883062    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.911503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.910766    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.921587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.921551    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlzxr\" (UniqueName: \"kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-kube-api-access-qlzxr\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:20.937328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:20.937295    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:21.126668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:21.126634    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"]
Nov 03 20:01:21.217063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:21.216994    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:21.236100 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:21.236065    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:21.369329 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:21.369284    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:21.605711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:21.605621    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"]
Nov 03 20:01:21.615155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:21.614617    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:21.615155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:21.614687    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:21.615155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:21.614719    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:22.000433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:22.000402    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:22.005833 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:22.005795    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" event={"ID":"fe201d89-2f4b-47f8-8fa5-983a86e2f87f","Type":"ContainerStarted","Data":"db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541"}
Nov 03 20:01:22.006031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:22.005868    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" event={"ID":"fe201d89-2f4b-47f8-8fa5-983a86e2f87f","Type":"ContainerStarted","Data":"35bce39461f58bef9d89ea1e87dd754db1b0c7cc15cd7dec5437086f4e2241a0"}
Nov 03 20:01:22.032047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:22.031984    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" podStartSLOduration=2.031964416 podStartE2EDuration="2.031964416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:22.02974731 +0000 UTC m=+1434.184649730" watchObservedRunningTime="2026-11-03 20:01:22.031964416 +0000 UTC m=+1434.186866836"
Nov 03 20:01:22.141278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:22.141243    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:22.224163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:22.218658    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:22.224722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:22.224597    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:01:22.224722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:22.224659    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 20:01:22.571713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:22.571613    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:22.978533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:22.974903    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:23.218158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:23.218120    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:01:23.218605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:23.218182    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 20:01:23.422336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:23.422249    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:23.901258 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:23.901220    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:24.044972 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:24.044934    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:24.217385 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:24.217351    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:01:24.217615 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:24.217415    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 20:01:24.280188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:24.280159    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:25.222661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:25.222623    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:01:25.252613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:25.252564    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 20:01:26.144505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:26.143503    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:26.144505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:26.140258    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk" podUID="abcc86d4-6bdb-433d-9c41-f14f00985280" containerName="payload-processing" containerID="cri-o://dd8485b4faf08de9f501e353b9aebc1aa76e5ef0ce084ef9be53485fb26c9280" gracePeriod=25
Nov 03 20:01:26.221760 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:26.221720    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:01:26.222003 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:26.221976    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 20:01:26.283644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:26.283592    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:26.493645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:26.493583    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:26.588764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:26.588706    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:26.675508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:26.674597    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:26.804719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:26.804599    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:26.810662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:26.810616    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.093197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.093052    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.217919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.217878    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:01:27.222577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.222529    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 20:01:27.306634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.306578    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.307280 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.307256    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 20:01:27.464522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.462698    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.487514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.476534    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:27.511522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.503800    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.511522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.504554    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:01:27.606507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.583860    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.606507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.584907    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.606507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.585156    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Nov 03 20:01:27.609302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.607201    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.609302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.607296    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:01:27.609302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.607717    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:01:27.741565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.700605    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.818506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.806854    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:01:27.818506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.808764    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:01:27.818506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:27.808823    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 20:01:28.096576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.095237    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.256658 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.254684    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:01:28.256658 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.254742    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 20:01:28.333550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.312690    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.356718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.344238    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:01:28.568516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.568452    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:01:28.568718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.568534    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p" podUID="85068d81-d023-4e25-a6c1-3aff06dc9bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.580551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.571965    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.591731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.591679    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.593140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.593118    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:01:28.610507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.610398    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.629432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.629391    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.631225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.631199    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 20:01:28.808056 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.808005    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:01:28.810177 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.810129    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.991344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.991304    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:01:28.991580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:28.991552    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc" podUID="f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:29.231567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:29.229312    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:01:29.231567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:29.229384    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 20:01:29.232580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:29.232543    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:01:29.399647 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:29.349464    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 20:01:29.583745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:29.583637    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:29.632503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:29.597578    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:29.632503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:29.598384    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:01:29.661503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:29.649946    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:01:29.823506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:29.796631    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:29.823506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:29.816910    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:29.823506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:29.817242    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:29.907217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:29.907110    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:30.140563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.140521    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk" podUID="abcc86d4-6bdb-433d-9c41-f14f00985280" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:01:30.197561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.189107    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:30.244508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.220263    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:01:30.244508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.220335    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 20:01:30.269594 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.269563    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 20:01:30.271151 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.271133    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 20:01:30.404031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.403257    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:30.510698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.508323    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:01:30.600499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.598542    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:30.614501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.610575    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:01:30.614501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.612252    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:01:30.618413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.617706    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 20:01:30.648244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.648104    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:01:30.651394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.651374    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:01:30.658915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.658877    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:01:30.924495 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:30.924399    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:31.061805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:31.061741    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:31.218271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:31.218235    5659 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:01:31.218435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:31.218288    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 20:01:32.008250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:32.008215    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:32.334227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:32.334143    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:32.335221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:32.335143    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:32.335353 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:32.335248    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:01:32.516307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:32.516272    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:32.954598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:32.954554    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:33.782138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:33.782093    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:34.416793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:34.416723    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:34.546090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:34.546055    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:35.096049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:35.096017    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:35.952154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:35.952107    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:36.398771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:36.398742    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:36.528123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:36.528052    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:37.344640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:37.344608    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:37.551217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:37.551179    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:37.687763 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:37.687683    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:37.850080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:37.850047    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:38.167128 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:38.167096    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"]
Nov 03 20:01:38.167401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:38.167357    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" podUID="74da69e9-a168-4ae5-a882-daca94333466" containerName="istio-proxy" containerID="cri-o://855f4d3259192e4d9afefaaed5ed9b0941633808c49dc562d6e0ff01150f70de" gracePeriod=30
Nov 03 20:01:38.236449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:38.236413    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:38.731942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:38.731911    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:39.155925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:39.155848    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:39.298925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:39.298889    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:39.586972 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:39.586938    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:39.810546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:39.810465    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:40.091206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:40.091161    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk" podUID="abcc86d4-6bdb-433d-9c41-f14f00985280" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:01:40.155619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:40.155577    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:40.293739 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:40.293690    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:40.929727 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:40.929691    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:41.667024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:41.666995    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:41.799015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:41.798981    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:42.112537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:42.112476    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:42.504008 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:42.503968    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:42.737808 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:42.737770    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:43.482924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.482582    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 20:01:43.622757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.622722    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-istio-token\" (UniqueName: \"kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-istio-token\") pod \"74da69e9-a168-4ae5-a882-daca94333466\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") "
Nov 03 20:01:43.622927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.622792    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-envoy\") pod \"74da69e9-a168-4ae5-a882-daca94333466\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") "
Nov 03 20:01:43.622927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.622878    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-data\") pod \"74da69e9-a168-4ae5-a882-daca94333466\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") "
Nov 03 20:01:43.622927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.622900    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-certs\") pod \"74da69e9-a168-4ae5-a882-daca94333466\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") "
Nov 03 20:01:43.623123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.623003    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74da69e9-a168-4ae5-a882-daca94333466-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74da69e9-a168-4ae5-a882-daca94333466-istio-podinfo\") pod \"74da69e9-a168-4ae5-a882-daca94333466\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") "
Nov 03 20:01:43.623123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.623093    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-kube-api-access-n5k5l\" (UniqueName: \"kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-kube-api-access-n5k5l\") pod \"74da69e9-a168-4ae5-a882-daca94333466\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") "
Nov 03 20:01:43.623237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.623138    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-socket\") pod \"74da69e9-a168-4ae5-a882-daca94333466\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") "
Nov 03 20:01:43.623237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.623164    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-data" pod "74da69e9-a168-4ae5-a882-daca94333466" (UID: "74da69e9-a168-4ae5-a882-daca94333466"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:43.623237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.623180    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74da69e9-a168-4ae5-a882-daca94333466-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74da69e9-a168-4ae5-a882-daca94333466-istiod-ca-cert\") pod \"74da69e9-a168-4ae5-a882-daca94333466\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") "
Nov 03 20:01:43.623237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.623208    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-credential-socket\") pod \"74da69e9-a168-4ae5-a882-daca94333466\" (UID: \"74da69e9-a168-4ae5-a882-daca94333466\") "
Nov 03 20:01:43.623462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.623284    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-certs" pod "74da69e9-a168-4ae5-a882-daca94333466" (UID: "74da69e9-a168-4ae5-a882-daca94333466"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:43.623573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.623550    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-socket" pod "74da69e9-a168-4ae5-a882-daca94333466" (UID: "74da69e9-a168-4ae5-a882-daca94333466"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:43.623800 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.623772    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-credential-socket" pod "74da69e9-a168-4ae5-a882-daca94333466" (UID: "74da69e9-a168-4ae5-a882-daca94333466"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:43.624013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.623990    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:43.624101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.624046    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:43.624101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.624063    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:43.624513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.624465    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74da69e9-a168-4ae5-a882-daca94333466-istiod-ca-cert" pod "74da69e9-a168-4ae5-a882-daca94333466" (UID: "74da69e9-a168-4ae5-a882-daca94333466"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:43.626611 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.626589    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-envoy" pod "74da69e9-a168-4ae5-a882-daca94333466" (UID: "74da69e9-a168-4ae5-a882-daca94333466"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:43.626710 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.626587    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74da69e9-a168-4ae5-a882-daca94333466-istio-podinfo" pod "74da69e9-a168-4ae5-a882-daca94333466" (UID: "74da69e9-a168-4ae5-a882-daca94333466"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:01:43.626710 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.626638    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-kube-api-access-n5k5l" pod "74da69e9-a168-4ae5-a882-daca94333466" (UID: "74da69e9-a168-4ae5-a882-daca94333466"). InnerVolumeSpecName "kube-api-access-n5k5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:43.626805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.626657    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-istio-token" pod "74da69e9-a168-4ae5-a882-daca94333466" (UID: "74da69e9-a168-4ae5-a882-daca94333466"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:43.649736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.649693    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:43.725774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.725686    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74da69e9-a168-4ae5-a882-daca94333466-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:43.725774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.725715    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5k5l\" (UniqueName: \"kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-kube-api-access-n5k5l\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:43.725774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.725734    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74da69e9-a168-4ae5-a882-daca94333466-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:43.725774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.725747    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:43.725774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.725760    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74da69e9-a168-4ae5-a882-daca94333466-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:43.725774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.725772    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74da69e9-a168-4ae5-a882-daca94333466-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:43.971367 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:43.971298    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:44.100727 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:44.100694    5659 generic.go:356] "Generic (PLEG): container finished" podID="74da69e9-a168-4ae5-a882-daca94333466" containerID="855f4d3259192e4d9afefaaed5ed9b0941633808c49dc562d6e0ff01150f70de" exitCode=0
Nov 03 20:01:44.100913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:44.100759    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"
Nov 03 20:01:44.100913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:44.100784    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" event={"ID":"74da69e9-a168-4ae5-a882-daca94333466","Type":"ContainerDied","Data":"855f4d3259192e4d9afefaaed5ed9b0941633808c49dc562d6e0ff01150f70de"}
Nov 03 20:01:44.100913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:44.100841    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk" event={"ID":"74da69e9-a168-4ae5-a882-daca94333466","Type":"ContainerDied","Data":"72e87f9c7e4623787954d55e5886ac6f2a2c19de77c527ae5f9d1d9c5b130976"}
Nov 03 20:01:44.100913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:44.100864    5659 scope.go:122] "RemoveContainer" containerID="855f4d3259192e4d9afefaaed5ed9b0941633808c49dc562d6e0ff01150f70de"
Nov 03 20:01:44.110559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:44.110542    5659 scope.go:122] "RemoveContainer" containerID="96ee96b4fb32413f0f4a13240ff155b1f3e23752fb2bf428b899ad41fb553693"
Nov 03 20:01:44.120409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:44.120385    5659 scope.go:122] "RemoveContainer" containerID="855f4d3259192e4d9afefaaed5ed9b0941633808c49dc562d6e0ff01150f70de"
Nov 03 20:01:44.120626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:44.120610    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"]
Nov 03 20:01:44.120728 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:01:44.120708    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855f4d3259192e4d9afefaaed5ed9b0941633808c49dc562d6e0ff01150f70de\": container with ID starting with 855f4d3259192e4d9afefaaed5ed9b0941633808c49dc562d6e0ff01150f70de not found: ID does not exist" containerID="855f4d3259192e4d9afefaaed5ed9b0941633808c49dc562d6e0ff01150f70de"
Nov 03 20:01:44.120768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:44.120741    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855f4d3259192e4d9afefaaed5ed9b0941633808c49dc562d6e0ff01150f70de"} err="failed to get container status \"855f4d3259192e4d9afefaaed5ed9b0941633808c49dc562d6e0ff01150f70de\": rpc error: code = NotFound desc = could not find container \"855f4d3259192e4d9afefaaed5ed9b0941633808c49dc562d6e0ff01150f70de\": container with ID starting with 855f4d3259192e4d9afefaaed5ed9b0941633808c49dc562d6e0ff01150f70de not found: ID does not exist"
Nov 03 20:01:44.120768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:44.120763    5659 scope.go:122] "RemoveContainer" containerID="96ee96b4fb32413f0f4a13240ff155b1f3e23752fb2bf428b899ad41fb553693"
Nov 03 20:01:44.121017 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:01:44.120998    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ee96b4fb32413f0f4a13240ff155b1f3e23752fb2bf428b899ad41fb553693\": container with ID starting with 96ee96b4fb32413f0f4a13240ff155b1f3e23752fb2bf428b899ad41fb553693 not found: ID does not exist" containerID="96ee96b4fb32413f0f4a13240ff155b1f3e23752fb2bf428b899ad41fb553693"
Nov 03 20:01:44.121076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:44.121022    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ee96b4fb32413f0f4a13240ff155b1f3e23752fb2bf428b899ad41fb553693"} err="failed to get container status \"96ee96b4fb32413f0f4a13240ff155b1f3e23752fb2bf428b899ad41fb553693\": rpc error: code = NotFound desc = could not find container \"96ee96b4fb32413f0f4a13240ff155b1f3e23752fb2bf428b899ad41fb553693\": container with ID starting with 96ee96b4fb32413f0f4a13240ff155b1f3e23752fb2bf428b899ad41fb553693 not found: ID does not exist"
Nov 03 20:01:44.127513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:44.127474    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-1b1d04-openshift-default-6467df856f-txmdk"]
Nov 03 20:01:44.390677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:44.390603    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:45.037862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:45.037814    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:45.217628 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:01:45.217596    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:01:45.731682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:45.731652    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:45.969606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:45.969569    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74da69e9-a168-4ae5-a882-daca94333466" path="/var/lib/kubelet/pods/74da69e9-a168-4ae5-a882-daca94333466/volumes"
Nov 03 20:01:46.363067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:46.363036    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"
Nov 03 20:01:46.437154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:46.437073    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:46.466466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:46.466427    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abcc86d4-6bdb-433d-9c41-f14f00985280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcc86d4-6bdb-433d-9c41-f14f00985280-plugins-config-volume\") pod \"abcc86d4-6bdb-433d-9c41-f14f00985280\" (UID: \"abcc86d4-6bdb-433d-9c41-f14f00985280\") "
Nov 03 20:01:46.466815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:46.466789    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abcc86d4-6bdb-433d-9c41-f14f00985280-kube-api-access-sk9r6\" (UniqueName: \"kubernetes.io/projected/abcc86d4-6bdb-433d-9c41-f14f00985280-kube-api-access-sk9r6\") pod \"abcc86d4-6bdb-433d-9c41-f14f00985280\" (UID: \"abcc86d4-6bdb-433d-9c41-f14f00985280\") "
Nov 03 20:01:46.477502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:46.468291    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abcc86d4-6bdb-433d-9c41-f14f00985280-plugins-config-volume" pod "abcc86d4-6bdb-433d-9c41-f14f00985280" (UID: "abcc86d4-6bdb-433d-9c41-f14f00985280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:46.495162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:46.488064    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcc86d4-6bdb-433d-9c41-f14f00985280-kube-api-access-sk9r6" pod "abcc86d4-6bdb-433d-9c41-f14f00985280" (UID: "abcc86d4-6bdb-433d-9c41-f14f00985280"). InnerVolumeSpecName "kube-api-access-sk9r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:46.568795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:46.568752    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcc86d4-6bdb-433d-9c41-f14f00985280-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:46.568795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:46.568784    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk9r6\" (UniqueName: \"kubernetes.io/projected/abcc86d4-6bdb-433d-9c41-f14f00985280-kube-api-access-sk9r6\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:47.123977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:47.123947    5659 generic.go:356] "Generic (PLEG): container finished" podID="abcc86d4-6bdb-433d-9c41-f14f00985280" containerID="dd8485b4faf08de9f501e353b9aebc1aa76e5ef0ce084ef9be53485fb26c9280" exitCode=0
Nov 03 20:01:47.124196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:47.123996    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk" event={"ID":"abcc86d4-6bdb-433d-9c41-f14f00985280","Type":"ContainerDied","Data":"dd8485b4faf08de9f501e353b9aebc1aa76e5ef0ce084ef9be53485fb26c9280"}
Nov 03 20:01:47.124196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:47.123998    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"
Nov 03 20:01:47.124196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:47.124044    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk" event={"ID":"abcc86d4-6bdb-433d-9c41-f14f00985280","Type":"ContainerDied","Data":"bd01b4bef9053d732ff7a7c6eedcef21343d44a1dd45ddf15a582db2f345332e"}
Nov 03 20:01:47.124196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:47.124067    5659 scope.go:122] "RemoveContainer" containerID="dd8485b4faf08de9f501e353b9aebc1aa76e5ef0ce084ef9be53485fb26c9280"
Nov 03 20:01:47.130141 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:47.130105    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:47.193598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:47.190498    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"]
Nov 03 20:01:47.202440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:47.202380    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1b1d04-86f4c75484-c5fgk"]
Nov 03 20:01:47.206591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:47.206536    5659 scope.go:122] "RemoveContainer" containerID="dd8485b4faf08de9f501e353b9aebc1aa76e5ef0ce084ef9be53485fb26c9280"
Nov 03 20:01:47.206973 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:01:47.206953    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8485b4faf08de9f501e353b9aebc1aa76e5ef0ce084ef9be53485fb26c9280\": container with ID starting with dd8485b4faf08de9f501e353b9aebc1aa76e5ef0ce084ef9be53485fb26c9280 not found: ID does not exist" containerID="dd8485b4faf08de9f501e353b9aebc1aa76e5ef0ce084ef9be53485fb26c9280"
Nov 03 20:01:47.207038 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:47.206988    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8485b4faf08de9f501e353b9aebc1aa76e5ef0ce084ef9be53485fb26c9280"} err="failed to get container status \"dd8485b4faf08de9f501e353b9aebc1aa76e5ef0ce084ef9be53485fb26c9280\": rpc error: code = NotFound desc = could not find container \"dd8485b4faf08de9f501e353b9aebc1aa76e5ef0ce084ef9be53485fb26c9280\": container with ID starting with dd8485b4faf08de9f501e353b9aebc1aa76e5ef0ce084ef9be53485fb26c9280 not found: ID does not exist"
Nov 03 20:01:47.828621 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:47.828586    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:47.974916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:47.974878    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abcc86d4-6bdb-433d-9c41-f14f00985280" path="/var/lib/kubelet/pods/abcc86d4-6bdb-433d-9c41-f14f00985280/volumes"
Nov 03 20:01:48.600162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:48.600133    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:50.299622 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:50.299570    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:50.371676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:50.318710    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:50.475965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:50.433595    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:01:50.561553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:50.442136    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:50.629812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:50.617307    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:01:50.629812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:50.627580    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:01:50.633164 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:50.632418    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:50.663644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:50.663599    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:50.709932 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:50.685822    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:50.726582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:50.726547    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:50.822677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:50.815718    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:51.392709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.392647    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 20:01:51.396437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.396403    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:51.402172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.402151    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 20:01:51.402422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.402393    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 20:01:51.609822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.580102    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:51.609822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.580800    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:01:51.634225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.632588    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:51.634225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.633336    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 20:01:51.727365 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.709352    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:51.729199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.727670    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:01:51.730080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.727705    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:01:51.730834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.727766    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:51.746094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.729813    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:01:51.746776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.746757    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:01:51.746964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.746952    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:01:51.747099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.747087    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 20:01:51.822975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.822934    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:51.827044 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:51.827008    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:01:52.528878 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.528836    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:01:52.529529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.529476    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 20:01:52.529776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.529747    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:01:52.529990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.529961    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:01:52.530147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.529553    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:52.569217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.569185    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 20:01:52.578082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.578051    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:01:52.671665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.671593    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:52.673614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.673590    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:52.749595 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.749546    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:01:52.827039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.826936    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:52.853578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.853154    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:01:52.934060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.926644    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 20:01:52.965809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:52.964903    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 20:01:54.052696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:54.052658    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:54.835512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:54.835454    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:55.416036 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:55.416003    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:55.626506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:55.626444    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:56.624839 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:56.624804    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:56.913113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:56.913029    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:57.810673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:57.810642    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:57.948659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:57.948626    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:58.435680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:58.435648    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:58.573885 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:58.573825    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:58.877734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:58.877703    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:59.789402 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:01:59.789366    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:00.457536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:00.457502    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:01.147462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:01.147424    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:02.099996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:02.099954    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:02.236824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:02.236791    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:03.217237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:03.217204    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:03.950787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:03.950761    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:04.757093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:04.757059    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:05.210650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:05.210610    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"]
Nov 03 20:02:05.556963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:05.556894    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:05.779605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:05.779556    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp" podUID="26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" containerName="istio-proxy" containerID="cri-o://097e7cd7adc5884e46c4cadcb2ac1679fe3d07a55946e3686b643b95adace303" gracePeriod=30
Nov 03 20:02:06.241590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:06.241550    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:06.938137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:06.938066    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:07.626125 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:07.626097    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:08.651951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:08.651903    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:09.998503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:09.984281    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:10.761516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:10.761455    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:11.020666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.020583    5659 generic.go:356] "Generic (PLEG): container finished" podID="26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" containerID="097e7cd7adc5884e46c4cadcb2ac1679fe3d07a55946e3686b643b95adace303" exitCode=0
Nov 03 20:02:11.021030 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.020656    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp" event={"ID":"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad","Type":"ContainerDied","Data":"097e7cd7adc5884e46c4cadcb2ac1679fe3d07a55946e3686b643b95adace303"}
Nov 03 20:02:11.062802 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.062778    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 20:02:11.088587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.088551    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-token\" (UniqueName: \"kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-token\") pod \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") "
Nov 03 20:02:11.088776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.088658    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-data\") pod \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") "
Nov 03 20:02:11.088776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.088711    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-socket\") pod \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") "
Nov 03 20:02:11.088953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.088816    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istiod-ca-cert\") pod \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") "
Nov 03 20:02:11.088953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.088892    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-certs\") pod \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") "
Nov 03 20:02:11.088953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.088927    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-podinfo\") pod \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") "
Nov 03 20:02:11.089144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.088977    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-envoy\") pod \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") "
Nov 03 20:02:11.089144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.089021    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-credential-socket\") pod \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") "
Nov 03 20:02:11.089144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.089025    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-data" pod "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" (UID: "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:11.089144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.089102    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-kube-api-access-cfw4f\" (UniqueName: \"kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-kube-api-access-cfw4f\") pod \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\" (UID: \"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad\") "
Nov 03 20:02:11.090148 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.089134    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-socket" pod "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" (UID: "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:11.090148 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.089376    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istiod-ca-cert" pod "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" (UID: "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:11.090408 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.090384    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-credential-socket" pod "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" (UID: "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:11.090474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.090444    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-certs" pod "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" (UID: "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:11.091227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.090669    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:11.091227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.090696    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:11.091227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.090710    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:11.091227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.090722    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:11.091227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.090734    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:11.093074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.092582    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-envoy" pod "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" (UID: "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:11.093332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.093305    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-podinfo" pod "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" (UID: "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:11.093435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.093374    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-kube-api-access-cfw4f" pod "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" (UID: "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad"). InnerVolumeSpecName "kube-api-access-cfw4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:11.094917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.094896    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-token" pod "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" (UID: "26a5c8dc-4dfc-4108-bc92-88bc2ed23cad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:11.192558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.192527    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:11.192558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.192554    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:11.192558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.192564    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfw4f\" (UniqueName: \"kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-kube-api-access-cfw4f\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:11.192809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.192574    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:11.560394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:11.560353    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:12.028199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:12.028165    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"
Nov 03 20:02:12.028199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:12.028182    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp" event={"ID":"26a5c8dc-4dfc-4108-bc92-88bc2ed23cad","Type":"ContainerDied","Data":"58ddb86b5aa219672692b4d5d5674caf0d702892e5ffb48ea5e2d661ee3d8c96"}
Nov 03 20:02:12.028785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:12.028255    5659 scope.go:122] "RemoveContainer" containerID="097e7cd7adc5884e46c4cadcb2ac1679fe3d07a55946e3686b643b95adace303"
Nov 03 20:02:12.057733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:12.057703    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"]
Nov 03 20:02:12.062067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:12.062039    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-ktjlp"]
Nov 03 20:02:12.419713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:12.419638    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:13.243085 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:13.243048    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:13.957518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:13.951178    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:13.980827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:13.980777    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" path="/var/lib/kubelet/pods/26a5c8dc-4dfc-4108-bc92-88bc2ed23cad/volumes"
Nov 03 20:02:14.894908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:14.894877    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:15.216746 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:02:15.216713    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:02:15.517381 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:15.517308    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:15.838617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:15.838532    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:16.560069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:16.560034    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:16.742473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:16.742439    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:17.183978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:17.183941    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:17.318832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:17.318797    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:17.462503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:17.461371    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:18.114629 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:18.113677    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:18.945788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:18.945757    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:19.783149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:19.783118    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:20.478804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:20.478771    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:20.629619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:20.629568    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:21.546098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:21.546070    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:21.778103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:21.778053    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:21.956586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:21.945413    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:22.656444 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:22.656411    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:23.344907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:23.344872    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:24.129901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:24.129868    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:24.853590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:24.853553    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:25.523390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:25.523338    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:25.688176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:25.688131    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:26.534044 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:26.534011    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:28.927508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:28.925092    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:28.934047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:28.933999    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.143275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:29.143234    5659 kubelet.go:1689] "Image garbage collection succeeded"
Nov 03 20:02:29.566508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:29.565665    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.578506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:29.574777    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 20:02:29.604343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:29.597658    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.679620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:29.677846    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.799508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:29.797580    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.896428 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:29.896326    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.933705 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:29.933606    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.934467 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:29.934386    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.952568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:29.942580    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:02:29.952568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:29.942668    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 20:02:30.373507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:30.359140    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:02:30.373507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:30.360230    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:30.433619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:30.406909    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:30.587250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:30.587195    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 20:02:30.600707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:30.600646    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:30.651630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:30.630836    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:30.669588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:30.654261    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:02:30.730713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:30.730649    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:30.963557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:30.797856    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:30.988708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:30.975883    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:31.154140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.078691    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:31.177864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:30.967291    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:31.267557 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:02:31.265637    5659 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.304s"
Nov 03 20:02:31.267557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.265771    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 20:02:31.267557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.265832    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:02:31.267557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.265908    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:02:31.267557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.265942    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 20:02:31.267557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.265989    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 20:02:31.267557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.266025    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:02:31.594270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.593138    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 20:02:31.608993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.607239    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:31.655688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.654595    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:31.655688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.655012    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:02:31.698909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.698865    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:02:31.760943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.760905    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:31.804664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.803657    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:31.821764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:31.821740    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:02:32.025329 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:32.022001    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 20:02:32.027154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:32.026661    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:02:32.127874 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:32.127842    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 20:02:32.128153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:32.128134    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:02:32.128308 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:32.128294    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:02:32.128443 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:32.128430    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 20:02:32.128597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:32.128584    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 20:02:32.128725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:32.128714    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:02:33.433894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:33.433857    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:34.369838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:34.369801    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:35.204205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:35.204172    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:36.050725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:36.050689    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:36.462805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:36.462778    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:36.804180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:36.804093    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:37.530832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:37.530798    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:38.607130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:38.607096    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:39.354784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:39.354745    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:40.046021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:40.045987    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:40.690496 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:40.690453    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:40.954757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:40.954716    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:41.821150 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:41.821111    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:42.454123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:42.454090    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:42.618975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:42.618944    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:43.377733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:43.377692    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:44.267205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:44.267168    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:45.166318 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:45.166286    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:45.246099 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:02:45.246064    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:02:46.058666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:46.058599    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:49.224598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:49.207611    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:49.254805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:49.254769    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:49.614564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:49.610767    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:49.731956 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:49.731914    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:49.900828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:49.881324    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:50.018503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:50.017285    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:02:50.111599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:50.066579    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:50.111599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:50.068962    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:50.111599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:50.074897    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:50.263915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:50.256861    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:02:50.614057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:50.611044    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 20:02:50.614057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:50.611435    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 20:02:51.019078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.004541    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 20:02:51.033118 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.020646    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:51.054676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.046465    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:02:51.070587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.070529    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:02:51.072096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.072017    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:02:51.128650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.128600    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:51.192371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.192114    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:51.243306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.243277    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:02:51.257514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.257458    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:51.274094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.259329    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:02:51.274094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.273394    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:02:51.323249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.322778    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:02:51.486563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.486532    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:51.951497 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:51.947543    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:52.411431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:52.411376    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:53.359105 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:53.359002    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:54.420846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:54.420801    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:55.406249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:55.406195    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:55.755434 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:55.755395    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:56.051859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:56.051774    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:56.179838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:56.179799    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:56.563999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:56.563967    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:56.837111 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:56.836994    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:57.309135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:57.309104    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:57.504504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:57.494054    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:57.866281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:57.866245    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:58.502943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:58.502871    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:58.916294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:58.916189    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:59.949655 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:02:59.949623    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:00.495091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:00.495052    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"]
Nov 03 20:03:00.495315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:00.495288    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy" containerID="cri-o://cc5de8de49a6da238ce48e3bf5f8c610544109cd7bd2d5ce6f58c4646b18dfa5" gracePeriod=30
Nov 03 20:03:01.044541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:01.031424    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:01.476272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:01.476238    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:01.617886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:01.617851    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:02.136037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:02.136003    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:02.770336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:02.770297    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:02.924054 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:02.924020    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:03.387305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:03.387269    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:03.534432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:03.534394    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:03.677218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:03.677139    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:03.845993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:03.845957    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:04.568755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:04.568722    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:04.706743 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:04.706668    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:05.387183 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387150    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"]
Nov 03 20:03:05.387790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387771    5659 cpu_manager.go:414] "containerMap: removing container" podUID="74da69e9-a168-4ae5-a882-daca94333466" containerName="istio-proxy"
Nov 03 20:03:05.387790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387790    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74da69e9-a168-4ae5-a882-daca94333466" containerName="istio-proxy"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387814    5659 cpu_manager.go:414] "containerMap: removing container" podUID="74da69e9-a168-4ae5-a882-daca94333466" containerName="istio-proxy"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387823    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74da69e9-a168-4ae5-a882-daca94333466" containerName="istio-proxy"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387850    5659 cpu_manager.go:414] "containerMap: removing container" podUID="33f77ab6-e954-4124-bd4f-ea0891241124" containerName="cleanup"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387855    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f77ab6-e954-4124-bd4f-ea0891241124" containerName="cleanup"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387863    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8027f464-332c-430c-976d-22327de2da7a" containerName="cleanup"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387868    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8027f464-332c-430c-976d-22327de2da7a" containerName="cleanup"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387876    5659 cpu_manager.go:414] "containerMap: removing container" podUID="64d32f4c-445e-4aa4-8ecb-bacb6903c699" containerName="cleanup"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387881    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d32f4c-445e-4aa4-8ecb-bacb6903c699" containerName="cleanup"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387890    5659 cpu_manager.go:414] "containerMap: removing container" podUID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" containerName="cleanup"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387897    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" containerName="cleanup"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387907    5659 cpu_manager.go:414] "containerMap: removing container" podUID="26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" containerName="istio-proxy"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387912    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" containerName="istio-proxy"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387921    5659 cpu_manager.go:414] "containerMap: removing container" podUID="abcc86d4-6bdb-433d-9c41-f14f00985280" containerName="payload-processing"
Nov 03 20:03:05.387938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.387926    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abcc86d4-6bdb-433d-9c41-f14f00985280" containerName="payload-processing"
Nov 03 20:03:05.388561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.388011    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="74da69e9-a168-4ae5-a882-daca94333466" containerName="istio-proxy"
Nov 03 20:03:05.388561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.388022    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8027f464-332c-430c-976d-22327de2da7a" containerName="cleanup"
Nov 03 20:03:05.388561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.388032    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f77ab6-e954-4124-bd4f-ea0891241124" containerName="cleanup"
Nov 03 20:03:05.388561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.388039    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d32f4c-445e-4aa4-8ecb-bacb6903c699" containerName="cleanup"
Nov 03 20:03:05.388561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.388047    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="490ddd7a-bf10-40e2-bbaf-0eb0f8049c9c" containerName="cleanup"
Nov 03 20:03:05.388561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.388053    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a5c8dc-4dfc-4108-bc92-88bc2ed23cad" containerName="istio-proxy"
Nov 03 20:03:05.388561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.388061    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49ffc14-16a0-495d-954f-a524e0d4a81a" containerName="istio-proxy"
Nov 03 20:03:05.388561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.388068    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="abcc86d4-6bdb-433d-9c41-f14f00985280" containerName="payload-processing"
Nov 03 20:03:05.388561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.388079    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="74da69e9-a168-4ae5-a882-daca94333466" containerName="istio-proxy"
Nov 03 20:03:05.391662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.391641    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.415541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.415517    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"]
Nov 03 20:03:05.480838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.480798    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:05.500373 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.500334    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.500573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.500500    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.500573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.500538    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.500573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.500569    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.500722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.500603    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.500722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.500681    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.500722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.500719    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.500881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.500756    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkcv6\" (UniqueName: \"kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-kube-api-access-xkcv6\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.500881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.500835    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.601846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.601802    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.602271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.601883    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.602271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.601917    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkcv6\" (UniqueName: \"kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-kube-api-access-xkcv6\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.602271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.602006    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.602271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.602069    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.602271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.602143    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.602271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.602177    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.602271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.602212    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.602271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.602249    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.602713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.602439    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.602713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.602622    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.602819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.602802    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.602922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.602888    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.603082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.603058    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.604951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.604924    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.605235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.605213    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.616462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.616389    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.616696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.616671    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkcv6\" (UniqueName: \"kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-kube-api-access-xkcv6\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.714611 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.714080    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:05.770736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.770709    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 20:03:05.808957 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.808923    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-data\") pod \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") "
Nov 03 20:03:05.809119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.809103    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istiod-ca-cert\") pod \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") "
Nov 03 20:03:05.809205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.809189    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-kube-api-access-xlbbn\" (UniqueName: \"kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-kube-api-access-xlbbn\") pod \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") "
Nov 03 20:03:05.809263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.809250    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-token\" (UniqueName: \"kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-token\") pod \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") "
Nov 03 20:03:05.809320 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.809284    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-podinfo\") pod \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") "
Nov 03 20:03:05.809383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.809323    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-envoy\") pod \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") "
Nov 03 20:03:05.809383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.809357    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-socket\") pod \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") "
Nov 03 20:03:05.809476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.809449    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-credential-socket\") pod \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") "
Nov 03 20:03:05.809476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.809477    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-certs\") pod \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\" (UID: \"38cd3425-f497-4177-a1ae-3e8dd72a8c16\") "
Nov 03 20:03:05.817238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.813339    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-certs" pod "38cd3425-f497-4177-a1ae-3e8dd72a8c16" (UID: "38cd3425-f497-4177-a1ae-3e8dd72a8c16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:05.817238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.816132    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istiod-ca-cert" pod "38cd3425-f497-4177-a1ae-3e8dd72a8c16" (UID: "38cd3425-f497-4177-a1ae-3e8dd72a8c16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:05.817238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.816387    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-data" pod "38cd3425-f497-4177-a1ae-3e8dd72a8c16" (UID: "38cd3425-f497-4177-a1ae-3e8dd72a8c16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:05.817899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.817874    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-credential-socket" pod "38cd3425-f497-4177-a1ae-3e8dd72a8c16" (UID: "38cd3425-f497-4177-a1ae-3e8dd72a8c16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:05.818040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.817891    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-socket" pod "38cd3425-f497-4177-a1ae-3e8dd72a8c16" (UID: "38cd3425-f497-4177-a1ae-3e8dd72a8c16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:05.818134 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.818079    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-podinfo" pod "38cd3425-f497-4177-a1ae-3e8dd72a8c16" (UID: "38cd3425-f497-4177-a1ae-3e8dd72a8c16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:03:05.825805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.825763    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-envoy" pod "38cd3425-f497-4177-a1ae-3e8dd72a8c16" (UID: "38cd3425-f497-4177-a1ae-3e8dd72a8c16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:05.834112 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.833098    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-token" pod "38cd3425-f497-4177-a1ae-3e8dd72a8c16" (UID: "38cd3425-f497-4177-a1ae-3e8dd72a8c16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:05.847565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.839185    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-kube-api-access-xlbbn" pod "38cd3425-f497-4177-a1ae-3e8dd72a8c16" (UID: "38cd3425-f497-4177-a1ae-3e8dd72a8c16"). InnerVolumeSpecName "kube-api-access-xlbbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:05.913199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.913152    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlbbn\" (UniqueName: \"kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-kube-api-access-xlbbn\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:05.913199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.913194    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:05.913464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.913208    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:05.913464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.913221    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:05.913464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.913233    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:05.913464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.913244    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:05.913464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.913256    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:05.913464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.913267    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:05.913464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.913281    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38cd3425-f497-4177-a1ae-3e8dd72a8c16-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:05.914192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.913991    5659 generic.go:356] "Generic (PLEG): container finished" podID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerID="cc5de8de49a6da238ce48e3bf5f8c610544109cd7bd2d5ce6f58c4646b18dfa5" exitCode=0
Nov 03 20:03:05.914192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.914139    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" event={"ID":"38cd3425-f497-4177-a1ae-3e8dd72a8c16","Type":"ContainerDied","Data":"cc5de8de49a6da238ce48e3bf5f8c610544109cd7bd2d5ce6f58c4646b18dfa5"}
Nov 03 20:03:05.915430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.914224    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr" event={"ID":"38cd3425-f497-4177-a1ae-3e8dd72a8c16","Type":"ContainerDied","Data":"69988785659dd1f6bde174e1748cd1511fda6693cff00720bdf87bc68f61ca60"}
Nov 03 20:03:05.915430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.914258    5659 scope.go:122] "RemoveContainer" containerID="cc5de8de49a6da238ce48e3bf5f8c610544109cd7bd2d5ce6f58c4646b18dfa5"
Nov 03 20:03:05.915430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.914454    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"
Nov 03 20:03:05.927747 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.927675    5659 scope.go:122] "RemoveContainer" containerID="cc5de8de49a6da238ce48e3bf5f8c610544109cd7bd2d5ce6f58c4646b18dfa5"
Nov 03 20:03:05.928392 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:03:05.928363    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc5de8de49a6da238ce48e3bf5f8c610544109cd7bd2d5ce6f58c4646b18dfa5\": container with ID starting with cc5de8de49a6da238ce48e3bf5f8c610544109cd7bd2d5ce6f58c4646b18dfa5 not found: ID does not exist" containerID="cc5de8de49a6da238ce48e3bf5f8c610544109cd7bd2d5ce6f58c4646b18dfa5"
Nov 03 20:03:05.928532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.928401    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc5de8de49a6da238ce48e3bf5f8c610544109cd7bd2d5ce6f58c4646b18dfa5"} err="failed to get container status \"cc5de8de49a6da238ce48e3bf5f8c610544109cd7bd2d5ce6f58c4646b18dfa5\": rpc error: code = NotFound desc = could not find container \"cc5de8de49a6da238ce48e3bf5f8c610544109cd7bd2d5ce6f58c4646b18dfa5\": container with ID starting with cc5de8de49a6da238ce48e3bf5f8c610544109cd7bd2d5ce6f58c4646b18dfa5 not found: ID does not exist"
Nov 03 20:03:05.943790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.943766    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"]
Nov 03 20:03:05.946096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.946069    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-6vlsr"]
Nov 03 20:03:05.970882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:05.970847    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" path="/var/lib/kubelet/pods/38cd3425-f497-4177-a1ae-3e8dd72a8c16/volumes"
Nov 03 20:03:06.037080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:06.037050    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"]
Nov 03 20:03:06.039389 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:03:06.039357    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a6df1c_1558_4b8c_b588_59b76dc7ffdd.slice/crio-aa2bc2318b3fca8ba7f76748093f5dbfc13fbc59ab2fa099a822cc8230e5d051 WatchSource:0}: Error finding container aa2bc2318b3fca8ba7f76748093f5dbfc13fbc59ab2fa099a822cc8230e5d051: Status 404 returned error can't find the container with id aa2bc2318b3fca8ba7f76748093f5dbfc13fbc59ab2fa099a822cc8230e5d051
Nov 03 20:03:06.041597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:06.041564    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:06.041720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:06.041654    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:06.041720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:06.041699    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:06.229471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:06.229439    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:06.923242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:06.923154    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" event={"ID":"09a6df1c-1558-4b8c-b588-59b76dc7ffdd","Type":"ContainerStarted","Data":"5fb192bae447d3fa54f9532ef5461e720b84752672a1e0c949d5b1988e2a88fe"}
Nov 03 20:03:06.923242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:06.923205    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" event={"ID":"09a6df1c-1558-4b8c-b588-59b76dc7ffdd","Type":"ContainerStarted","Data":"aa2bc2318b3fca8ba7f76748093f5dbfc13fbc59ab2fa099a822cc8230e5d051"}
Nov 03 20:03:06.968142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:06.968073    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" podStartSLOduration=1.968053958 podStartE2EDuration="1.968053958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:06.951405702 +0000 UTC m=+1539.106308122" watchObservedRunningTime="2026-11-03 20:03:06.968053958 +0000 UTC m=+1539.122956378"
Nov 03 20:03:06.989504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:06.986944    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:07.718503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:07.714877    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:07.729502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:07.721997    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 20:03:07.729502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:07.722058    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 20:03:08.192557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:08.192113    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:08.726995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:08.726952    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 20:03:08.744645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:08.744588    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 20:03:08.830564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:08.817598    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:09.035560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:09.034617    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:03:09.255685 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:09.255628    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:09.287039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:09.286922    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:09.336517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:09.333580    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:09.719519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:09.716687    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 20:03:09.719519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:09.716753    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 20:03:09.821518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:09.819570    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:09.821518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:09.820188    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 20:03:10.004557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.002571    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.123520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.121124    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.245559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.204006    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:03:10.252911 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.252857    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.268697 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.268614    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:03:10.288536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.288498    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.290519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.289417    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:03:10.310550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.309795    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.334528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.310995    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:03:10.337783 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.337750    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.398508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.373382    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:03:10.699560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.675050    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:10.733504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.728361    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 20:03:10.733504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.728437    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 20:03:10.919735 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:10.917581    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:11.098539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.094186    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:11.127827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.116616    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:11.127827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.116771    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 20:03:11.127827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.126803    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:11.139674 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.128668    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:03:11.237104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.237063    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:11.237581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.237554    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:03:11.255622 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.255569    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:11.256342 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.256319    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 20:03:11.322313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.322267    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:11.399997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.382748    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:11.399997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.383784    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:11.556259 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.556225    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:11.847507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.844087    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 20:03:11.847507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.844139    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 20:03:11.900532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.884264    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:11.921645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:11.921600    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:03:12.049617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.046710    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:03:12.094504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.094450    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 20:03:12.105714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.105624    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:12.133641 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.133590    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:12.237818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.237774    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:12.296021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.295975    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:03:12.338034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.337013    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:12.387279 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.337627    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:12.387565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.385575    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:12.569763 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.569718    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:12.703012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.702926    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 20:03:12.719880 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.719847    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 20:03:12.720173 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.720144    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 20:03:12.813444 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.813407    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:12.972577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:12.972537    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:13.100326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.100219    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:13.130690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.130637    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:13.145617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.145570    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:13.236143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.236059    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 20:03:13.317805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.317775    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:03:13.320184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.320152    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:03:13.343092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.343050    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:03:13.381507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.369958    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:13.382074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.382049    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:03:13.483991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.483944    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:03:13.620714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.620628    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:03:13.719942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.718903    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 20:03:13.719942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.718974    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 20:03:13.833769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.833736    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:03:13.910561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:13.909913    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 20:03:14.076363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:14.076248    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:14.257514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:14.256985    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:14.670437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:14.670345    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:14.715932 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:14.715889    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 20:03:14.716356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:14.715955    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 20:03:14.802954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:14.802924    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:15.292990 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:03:15.292916    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:03:15.550400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:15.550254    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:15.715176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:15.715142    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 20:03:15.715700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:15.715194    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 20:03:16.931758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:16.931396    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:16.932593 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:16.932572    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:16.939072 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:16.939052    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:03:16.971235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:16.971203    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:17.623598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:17.623555    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:17.753236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:17.753197    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:18.032993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:18.032961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:18.235585 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:18.235541    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:19.042208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:19.042177    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:19.195767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:19.195715    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:19.676323 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:19.676280    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:20.412266 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:20.412213    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:20.636973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:20.636922    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:21.151362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:21.151313    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:21.328923 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:21.328887    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:21.518027 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:21.517987    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:21.784138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:21.784058    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:22.405031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:22.405000    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:22.837826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:22.837786    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:23.833184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:23.833123    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:24.544013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:24.543979    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:25.424819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:25.424787    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:26.150954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:26.150918    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:26.279545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:26.279511    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:26.413588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:26.413470    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:26.561418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:26.561384    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:26.703777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:26.703698    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:27.215529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:27.215468    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:27.397257 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:27.397221    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:27.633018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:27.632933    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:27.761791 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:27.761759    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:27.890086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:27.889979    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:28.455506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:28.451194    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:29.133288 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:29.133247    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:29.270043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:29.270005    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:29.404852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:29.404774    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:30.337506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:30.332734    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:30.995790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:30.995758    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:31.132711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:31.132681    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:31.837743 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:31.837696    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:32.274981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:32.274911    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:32.413201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:32.413160    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:32.555357 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:32.555259    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:34.140650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:34.140603    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:34.167505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:34.153575    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:34.275258 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:34.275216    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:34.319380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:34.319322    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:34.386742 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:34.386693    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:34.392725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:34.392635    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:34.491430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:34.491385    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:35.106461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:35.106409    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:35.153663 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:35.153618    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:03:35.155017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:35.154996    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 20:03:35.207506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:35.203801    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:35.217351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:35.217319    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:03:35.350553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:35.343597    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:35.361571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:35.347682    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:35.409988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:35.361935    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:03:35.475709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:35.472694    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 20:03:35.532921 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:35.532894    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:03:35.539127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:35.535415    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:03:35.669983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:35.669861    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:36.545962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:36.545925    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:36.731376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:36.731345    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:37.710328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:37.710290    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:38.474815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:38.474782    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:38.662268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:38.662227    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:38.866856 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:38.866771    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:39.756746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:39.756710    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:40.229566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:40.229535    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:40.411364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:40.408049    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:40.666879 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:40.666783    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:41.221988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:41.221946    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:42.030780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:42.030747    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:43.019128 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:43.019095    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:43.152237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:43.152199    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:43.280894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:43.280799    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:43.670562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:43.667863    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:44.056610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:44.056577    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:44.222505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:44.219026    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:44.562214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:44.562176    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:44.849881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:44.849794    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:45.007354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:45.007314    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:45.142667 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:45.142585    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:45.422901 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:03:45.421177    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:03:45.844747 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:45.844716    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:47.013764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:47.013728    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:47.495753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:47.495722    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:47.626879 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:47.626845    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:48.106390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:48.106362    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:48.505830 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:48.505792    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:48.643510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:48.643449    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:48.785714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:48.785623    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:48.937920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:48.937881    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:49.692068 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.692030    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"]
Nov 03 20:03:49.693514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.693463    5659 cpu_manager.go:414] "containerMap: removing container" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy"
Nov 03 20:03:49.693514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.693508    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy"
Nov 03 20:03:49.693728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.693682    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="38cd3425-f497-4177-a1ae-3e8dd72a8c16" containerName="istio-proxy"
Nov 03 20:03:49.697142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.697118    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:49.701351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.701031    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-88a18e-serving-cert\""
Nov 03 20:03:49.707083 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.707058    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"]
Nov 03 20:03:49.726138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.726106    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:49.726288 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.726253    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:49.726429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.726408    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:49.726523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.726466    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwnsc\" (UniqueName: \"kubernetes.io/projected/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-kube-api-access-pwnsc\") pod \"maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:49.774678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.774650    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:49.831306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.831258    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:49.832646 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:03:49.832618    5659 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-serving-cert: secret "maas-api-e2e-shared-b-w4-88a18e-serving-cert" not found
Nov 03 20:03:49.832902 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:03:49.832884    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-maas-api-tls podName:99bedd28-f7f9-4445-9bf5-56f3a82b1df8 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:50.332859346 +0000 UTC m=+1582.487761762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-maas-api-tls") pod "maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt" (UID: "99bedd28-f7f9-4445-9bf5-56f3a82b1df8") : secret "maas-api-e2e-shared-b-w4-88a18e-serving-cert" not found
Nov 03 20:03:49.836765 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.836737    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwnsc\" (UniqueName: \"kubernetes.io/projected/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-kube-api-access-pwnsc\") pod \"maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:49.837152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.837112    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:49.837273 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.837209    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:49.838333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.838304    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:49.842162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.842135    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:49.855526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.852939    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwnsc\" (UniqueName: \"kubernetes.io/projected/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-kube-api-access-pwnsc\") pod \"maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:49.927859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:49.927827    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:50.157506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.154527    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:50.291356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.291238    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:50.335556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.332669    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"]
Nov 03 20:03:50.338289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.338252    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"
Nov 03 20:03:50.362531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.362389    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:50.372300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.364344    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-43c674-dockercfg-x5hkf\""
Nov 03 20:03:50.372300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.365334    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"]
Nov 03 20:03:50.372300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.367388    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:50.423462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.423363    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-43c674\""
Nov 03 20:03:50.468079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.468007    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:50.486510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.469901    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dq5l\" (UniqueName: \"kubernetes.io/projected/57bc1738-e920-4570-b31b-9e4937df03b0-kube-api-access-4dq5l\") pod \"payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm\" (UID: \"57bc1738-e920-4570-b31b-9e4937df03b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"
Nov 03 20:03:50.486867 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.486836    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57bc1738-e920-4570-b31b-9e4937df03b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm\" (UID: \"57bc1738-e920-4570-b31b-9e4937df03b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"
Nov 03 20:03:50.609506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.589378    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dq5l\" (UniqueName: \"kubernetes.io/projected/57bc1738-e920-4570-b31b-9e4937df03b0-kube-api-access-4dq5l\") pod \"payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm\" (UID: \"57bc1738-e920-4570-b31b-9e4937df03b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"
Nov 03 20:03:50.609506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.596639    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57bc1738-e920-4570-b31b-9e4937df03b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm\" (UID: \"57bc1738-e920-4570-b31b-9e4937df03b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"
Nov 03 20:03:50.609506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.597899    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57bc1738-e920-4570-b31b-9e4937df03b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm\" (UID: \"57bc1738-e920-4570-b31b-9e4937df03b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"
Nov 03 20:03:50.625619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.616383    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dq5l\" (UniqueName: \"kubernetes.io/projected/57bc1738-e920-4570-b31b-9e4937df03b0-kube-api-access-4dq5l\") pod \"payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm\" (UID: \"57bc1738-e920-4570-b31b-9e4937df03b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"
Nov 03 20:03:50.627592 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.627476    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:50.751919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:50.751879    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"
Nov 03 20:03:51.017669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:51.017635    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"]
Nov 03 20:03:51.018737 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:03:51.018701    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bedd28_f7f9_4445_9bf5_56f3a82b1df8.slice/crio-2c5507b8f2cb25af8f7883778fe6cfdfc9acb62cf71c2d63de21dd45ee213b36 WatchSource:0}: Error finding container 2c5507b8f2cb25af8f7883778fe6cfdfc9acb62cf71c2d63de21dd45ee213b36: Status 404 returned error can't find the container with id 2c5507b8f2cb25af8f7883778fe6cfdfc9acb62cf71c2d63de21dd45ee213b36
Nov 03 20:03:51.090058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:51.090029    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"]
Nov 03 20:03:51.091093 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:03:51.091062    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bc1738_e920_4570_b31b_9e4937df03b0.slice/crio-c620612e0d2249a30fea0d358b747ef55927aa5b02f49ba4c0a3e0b9d647aef7 WatchSource:0}: Error finding container c620612e0d2249a30fea0d358b747ef55927aa5b02f49ba4c0a3e0b9d647aef7: Status 404 returned error can't find the container with id c620612e0d2249a30fea0d358b747ef55927aa5b02f49ba4c0a3e0b9d647aef7
Nov 03 20:03:51.208689 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:51.208659    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:51.352556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:51.349564    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:51.381503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:51.378468    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt" event={"ID":"99bedd28-f7f9-4445-9bf5-56f3a82b1df8","Type":"ContainerStarted","Data":"2c5507b8f2cb25af8f7883778fe6cfdfc9acb62cf71c2d63de21dd45ee213b36"}
Nov 03 20:03:51.381503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:51.380679    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm" event={"ID":"57bc1738-e920-4570-b31b-9e4937df03b0","Type":"ContainerStarted","Data":"356242d5761fd5d4d8b9b58bb7d5f2fc8fc55c88f8007beaa21b9fe861afb516"}
Nov 03 20:03:51.381503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:51.380720    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm" event={"ID":"57bc1738-e920-4570-b31b-9e4937df03b0","Type":"ContainerStarted","Data":"c620612e0d2249a30fea0d358b747ef55927aa5b02f49ba4c0a3e0b9d647aef7"}
Nov 03 20:03:51.381503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:51.380900    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"
Nov 03 20:03:51.403110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:51.403059    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm" podStartSLOduration=1.403045269 podStartE2EDuration="1.403045269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:51.399736156 +0000 UTC m=+1583.554638575" watchObservedRunningTime="2026-11-03 20:03:51.403045269 +0000 UTC m=+1583.557947688"
Nov 03 20:03:51.489523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:51.489474    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:52.442628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:52.442372    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt" event={"ID":"99bedd28-f7f9-4445-9bf5-56f3a82b1df8","Type":"ContainerStarted","Data":"abbcb0ec5a14fe2e557b400568277b1192128ef7317fb7a1a1e97a6bc61f8945"}
Nov 03 20:03:52.442628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:52.442461    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:52.486238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:52.483094    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt" podStartSLOduration=3.051639205 podStartE2EDuration="3.483077748s" totalImagesPullingTime="431.438543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:52.482761965 +0000 UTC m=+1584.637664414" watchObservedRunningTime="2026-11-03 20:03:52.483077748 +0000 UTC m=+1584.637980168"
Nov 03 20:03:52.550498 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:52.550450    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:53.368186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:53.368150    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:54.130060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:54.130025    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:54.832356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:54.832320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:55.638761 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:55.638729    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:56.949892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:56.949587    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:57.930554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:57.930522    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:58.456535 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:58.456504    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:03:58.851791 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:58.851700    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:59.792949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:59.792915    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:59.932210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:03:59.932177    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:00.709396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:00.709365    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:00.915773 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:00.915741    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:01.331261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:01.331231    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:01.570846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:01.570810    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:01.750827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:01.750788    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:01.875061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:01.875031    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:02.445706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:02.445672    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"
Nov 03 20:04:02.574207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:02.574161    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:02.976505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:02.965652    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:03.093793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:03.093763    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:03.603110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:03.603076    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:03.919320 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:03.919229    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:04.604216 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:04.604180    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:04.789079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:04.789046    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:05.012039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:05.011991    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:05.881229 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:05.881186    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:05.910440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:05.910409    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"]
Nov 03 20:04:05.914857 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:05.914834    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"
Nov 03 20:04:05.924348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:05.924323    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-78d47a\""
Nov 03 20:04:05.926394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:05.926369    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"]
Nov 03 20:04:05.934374 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:05.934352    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-78d47a-dockercfg-47crr\""
Nov 03 20:04:05.973749 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:05.973589    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"]
Nov 03 20:04:05.984192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:05.984167    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"]
Nov 03 20:04:05.984445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:05.984286    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"
Nov 03 20:04:06.079507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.060406    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8n8\" (UniqueName: \"kubernetes.io/projected/49286968-069b-4864-a7bc-aedbb7d8a258-kube-api-access-zn8n8\") pod \"payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj\" (UID: \"49286968-069b-4864-a7bc-aedbb7d8a258\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"
Nov 03 20:04:06.079507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.060622    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49286968-069b-4864-a7bc-aedbb7d8a258-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj\" (UID: \"49286968-069b-4864-a7bc-aedbb7d8a258\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"
Nov 03 20:04:06.132845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.132045    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:06.171404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.169637    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8n8\" (UniqueName: \"kubernetes.io/projected/49286968-069b-4864-a7bc-aedbb7d8a258-kube-api-access-zn8n8\") pod \"payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj\" (UID: \"49286968-069b-4864-a7bc-aedbb7d8a258\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"
Nov 03 20:04:06.171404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.169766    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f623bbd-bc79-4756-b7fb-f4d656ebddff-plugins-config-volume\") pod \"payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf\" (UID: \"3f623bbd-bc79-4756-b7fb-f4d656ebddff\") " pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"
Nov 03 20:04:06.171404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.169835    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49286968-069b-4864-a7bc-aedbb7d8a258-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj\" (UID: \"49286968-069b-4864-a7bc-aedbb7d8a258\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"
Nov 03 20:04:06.171404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.169920    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68m56\" (UniqueName: \"kubernetes.io/projected/3f623bbd-bc79-4756-b7fb-f4d656ebddff-kube-api-access-68m56\") pod \"payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf\" (UID: \"3f623bbd-bc79-4756-b7fb-f4d656ebddff\") " pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"
Nov 03 20:04:06.171404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.171352    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49286968-069b-4864-a7bc-aedbb7d8a258-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj\" (UID: \"49286968-069b-4864-a7bc-aedbb7d8a258\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"
Nov 03 20:04:06.200663 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.200628    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8n8\" (UniqueName: \"kubernetes.io/projected/49286968-069b-4864-a7bc-aedbb7d8a258-kube-api-access-zn8n8\") pod \"payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj\" (UID: \"49286968-069b-4864-a7bc-aedbb7d8a258\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"
Nov 03 20:04:06.259428 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.253823    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"
Nov 03 20:04:06.274908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.274876    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f623bbd-bc79-4756-b7fb-f4d656ebddff-plugins-config-volume\") pod \"payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf\" (UID: \"3f623bbd-bc79-4756-b7fb-f4d656ebddff\") " pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"
Nov 03 20:04:06.275183 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.275158    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68m56\" (UniqueName: \"kubernetes.io/projected/3f623bbd-bc79-4756-b7fb-f4d656ebddff-kube-api-access-68m56\") pod \"payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf\" (UID: \"3f623bbd-bc79-4756-b7fb-f4d656ebddff\") " pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"
Nov 03 20:04:06.275763 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.275732    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f623bbd-bc79-4756-b7fb-f4d656ebddff-plugins-config-volume\") pod \"payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf\" (UID: \"3f623bbd-bc79-4756-b7fb-f4d656ebddff\") " pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"
Nov 03 20:04:06.290537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.290474    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68m56\" (UniqueName: \"kubernetes.io/projected/3f623bbd-bc79-4756-b7fb-f4d656ebddff-kube-api-access-68m56\") pod \"payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf\" (UID: \"3f623bbd-bc79-4756-b7fb-f4d656ebddff\") " pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"
Nov 03 20:04:06.420991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.420903    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"
Nov 03 20:04:06.558698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.558633    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"]
Nov 03 20:04:06.565476 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:04:06.564833    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49286968_069b_4864_a7bc_aedbb7d8a258.slice/crio-ae7174de33746de9b34fc36491a6525510105a649d5b8ecb8ae311296676c909 WatchSource:0}: Error finding container ae7174de33746de9b34fc36491a6525510105a649d5b8ecb8ae311296676c909: Status 404 returned error can't find the container with id ae7174de33746de9b34fc36491a6525510105a649d5b8ecb8ae311296676c909
Nov 03 20:04:06.715651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.715621    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"]
Nov 03 20:04:06.717093 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:04:06.717060    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f623bbd_bc79_4756_b7fb_f4d656ebddff.slice/crio-c21ca159863451c15c81be214fe176b5af5b7376ac0b58cfcdbea0e524bc404f WatchSource:0}: Error finding container c21ca159863451c15c81be214fe176b5af5b7376ac0b58cfcdbea0e524bc404f: Status 404 returned error can't find the container with id c21ca159863451c15c81be214fe176b5af5b7376ac0b58cfcdbea0e524bc404f
Nov 03 20:04:06.928440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:06.928396    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:07.560818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:07.560764    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj" event={"ID":"49286968-069b-4864-a7bc-aedbb7d8a258","Type":"ContainerStarted","Data":"8e154b611c9aa565536d5612f4efdff490b5ac936639a72999f7726f1fa87789"}
Nov 03 20:04:07.561002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:07.560869    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj" event={"ID":"49286968-069b-4864-a7bc-aedbb7d8a258","Type":"ContainerStarted","Data":"ae7174de33746de9b34fc36491a6525510105a649d5b8ecb8ae311296676c909"}
Nov 03 20:04:07.561002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:07.560902    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"
Nov 03 20:04:07.562295 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:07.562269    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf" event={"ID":"3f623bbd-bc79-4756-b7fb-f4d656ebddff","Type":"ContainerStarted","Data":"ab79d9804869c9cd74535af043c1601ab08642e5972f2b7b5ef50ba21bf66434"}
Nov 03 20:04:07.562408 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:07.562309    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf" event={"ID":"3f623bbd-bc79-4756-b7fb-f4d656ebddff","Type":"ContainerStarted","Data":"c21ca159863451c15c81be214fe176b5af5b7376ac0b58cfcdbea0e524bc404f"}
Nov 03 20:04:07.562449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:07.562416    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"
Nov 03 20:04:07.582071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:07.582015    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj" podStartSLOduration=2.581998201 podStartE2EDuration="2.581998201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:07.579375126 +0000 UTC m=+1599.734277539" watchObservedRunningTime="2026-11-03 20:04:07.581998201 +0000 UTC m=+1599.736900620"
Nov 03 20:04:07.600160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:07.600115    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf" podStartSLOduration=2.6001015929999998 podStartE2EDuration="2.600101593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:07.597277809 +0000 UTC m=+1599.752180230" watchObservedRunningTime="2026-11-03 20:04:07.600101593 +0000 UTC m=+1599.755004011"
Nov 03 20:04:07.771187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:07.771145    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:08.432402 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:08.432343    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:08.905599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:08.905563    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:09.173911 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:09.173829    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:09.923150 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:09.923116    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:10.723626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:10.723591    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:11.217264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.217233    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"]
Nov 03 20:04:11.221501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.221460    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.230528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.229279    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-bfc240-openshift-default-dockercfg-dxqxq\""
Nov 03 20:04:11.238130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.238105    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"]
Nov 03 20:04:11.374633 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.374595    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033f77a2-caed-43b4-8c63-30248e679ea2-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.374828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.374727    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.374879 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.374830    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.374966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.374941    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.375009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.374982    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.375127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.375107    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033f77a2-caed-43b4-8c63-30248e679ea2-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.375176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.375159    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.375297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.375277    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr6mt\" (UniqueName: \"kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-kube-api-access-dr6mt\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.375375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.375358    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.476953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.476857    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.477130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.477009    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033f77a2-caed-43b4-8c63-30248e679ea2-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.477130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.477047    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.477247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.477127    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.477247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.477174    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.477247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.477204    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.477405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.477268    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033f77a2-caed-43b4-8c63-30248e679ea2-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.477405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.477313    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.477405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.477326    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.477405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.477359    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr6mt\" (UniqueName: \"kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-kube-api-access-dr6mt\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.477931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.477720    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.478067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.478046    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.478244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.478227    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.478324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.478257    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033f77a2-caed-43b4-8c63-30248e679ea2-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.479929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.479903    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.480061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.479988    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033f77a2-caed-43b4-8c63-30248e679ea2-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.490503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.490461    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.497414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.497392    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr6mt\" (UniqueName: \"kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-kube-api-access-dr6mt\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.539654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.539624    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:11.581555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.580217    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:11.830602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.830529    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:11.862664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.862639    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"]
Nov 03 20:04:11.863676 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:04:11.863648    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033f77a2_caed_43b4_8c63_30248e679ea2.slice/crio-b53a819b2741f200c8874d296e7fd711df468647cbc86a7ada08f6f729d573ad WatchSource:0}: Error finding container b53a819b2741f200c8874d296e7fd711df468647cbc86a7ada08f6f729d573ad: Status 404 returned error can't find the container with id b53a819b2741f200c8874d296e7fd711df468647cbc86a7ada08f6f729d573ad
Nov 03 20:04:11.866408 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.866371    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:11.866531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.866449    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:11.866605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:11.866528    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:12.544820 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:12.544790    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:12.614160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:12.614120    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6" event={"ID":"033f77a2-caed-43b4-8c63-30248e679ea2","Type":"ContainerStarted","Data":"8d020de094144ab7f0abf14cee2895b13fe7608593f314d1471f5d86e5d77f35"}
Nov 03 20:04:12.615208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:12.614191    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6" event={"ID":"033f77a2-caed-43b4-8c63-30248e679ea2","Type":"ContainerStarted","Data":"b53a819b2741f200c8874d296e7fd711df468647cbc86a7ada08f6f729d573ad"}
Nov 03 20:04:12.640688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:12.640625    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6" podStartSLOduration=1.640604789 podStartE2EDuration="1.640604789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:12.637950517 +0000 UTC m=+1604.792852938" watchObservedRunningTime="2026-11-03 20:04:12.640604789 +0000 UTC m=+1604.795507208"
Nov 03 20:04:13.555508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:13.553183    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:13.555508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:13.553316    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 20:04:13.555508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:13.553357    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6" podUID="033f77a2-caed-43b4-8c63-30248e679ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 20:04:13.715163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:13.715131    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:14.547881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:14.547843    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 20:04:14.548167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:14.548137    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6" podUID="033f77a2-caed-43b4-8c63-30248e679ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 20:04:14.569708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:14.569679    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:15.404515 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:04:15.400689    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:04:15.487332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:15.478739    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:15.558512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:15.546906    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 20:04:15.558512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:15.546974    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6" podUID="033f77a2-caed-43b4-8c63-30248e679ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 20:04:16.404505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.403850    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"]
Nov 03 20:04:16.420455 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.420424    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"]
Nov 03 20:04:16.420796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.420737    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:16.426984 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.426954    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-78d47a-serving-cert\""
Nov 03 20:04:16.485504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.481354    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:16.526077 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.521050    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-metrics-certs\") pod \"maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:16.526077 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.521135    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpkh9\" (UniqueName: \"kubernetes.io/projected/a80a4e5e-576a-4933-9e86-bb68d58cd9db-kube-api-access-hpkh9\") pod \"maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:16.526077 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.521178    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80a4e5e-576a-4933-9e86-bb68d58cd9db-openshift-service-ca\") pod \"maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:16.526077 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.521279    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-maas-api-tls\") pod \"maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:16.575503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.575440    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 20:04:16.575711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.575530    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6" podUID="033f77a2-caed-43b4-8c63-30248e679ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 20:04:16.613656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.613623    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:16.622636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.622601    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-metrics-certs\") pod \"maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:16.622807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.622692    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpkh9\" (UniqueName: \"kubernetes.io/projected/a80a4e5e-576a-4933-9e86-bb68d58cd9db-kube-api-access-hpkh9\") pod \"maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:16.622807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.622753    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80a4e5e-576a-4933-9e86-bb68d58cd9db-openshift-service-ca\") pod \"maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:16.623221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.623198    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-maas-api-tls\") pod \"maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:16.623387 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:04:16.623372    5659 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-serving-cert: secret "maas-api-e2e-explicit-78d47a-serving-cert" not found
Nov 03 20:04:16.623451 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:04:16.623442    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-maas-api-tls podName:a80a4e5e-576a-4933-9e86-bb68d58cd9db nodeName:}" failed. No retries permitted until 2026-11-03 20:04:17.12342545 +0000 UTC m=+1609.278327861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-maas-api-tls") pod "maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4" (UID: "a80a4e5e-576a-4933-9e86-bb68d58cd9db") : secret "maas-api-e2e-explicit-78d47a-serving-cert" not found
Nov 03 20:04:16.623593 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.623571    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80a4e5e-576a-4933-9e86-bb68d58cd9db-openshift-service-ca\") pod \"maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:16.625167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.625147    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-metrics-certs\") pod \"maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:16.682278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.682201    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpkh9\" (UniqueName: \"kubernetes.io/projected/a80a4e5e-576a-4933-9e86-bb68d58cd9db-kube-api-access-hpkh9\") pod \"maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:16.764506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:16.757573    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:17.164772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:17.161835    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-maas-api-tls\") pod \"maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:17.200504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:17.200212    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-maas-api-tls\") pod \"maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:17.261322 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:17.261290    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:17.411598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:17.399718    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:17.535505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:17.532317    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:17.544507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:17.543583    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 20:04:17.544507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:17.543638    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6" podUID="033f77a2-caed-43b4-8c63-30248e679ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 20:04:17.745662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:17.745124    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:17.919258 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:17.919228    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:18.160883 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:18.160797    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"]
Nov 03 20:04:18.181856 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:04:18.181818    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80a4e5e_576a_4933_9e86_bb68d58cd9db.slice/crio-287c8ec8b8260a00da729884c8af6143fad9cd7cb01be5efb9cea7bdc71ced76 WatchSource:0}: Error finding container 287c8ec8b8260a00da729884c8af6143fad9cd7cb01be5efb9cea7bdc71ced76: Status 404 returned error can't find the container with id 287c8ec8b8260a00da729884c8af6143fad9cd7cb01be5efb9cea7bdc71ced76
Nov 03 20:04:18.183563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:18.183539    5659 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:04:18.555445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:18.552453    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 20:04:18.555445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:18.552599    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6" podUID="033f77a2-caed-43b4-8c63-30248e679ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 20:04:18.574306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:18.574271    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"
Nov 03 20:04:18.574540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:18.574400    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"
Nov 03 20:04:18.648714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:18.648678    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:18.767233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:18.766516    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4" event={"ID":"a80a4e5e-576a-4933-9e86-bb68d58cd9db","Type":"ContainerStarted","Data":"287c8ec8b8260a00da729884c8af6143fad9cd7cb01be5efb9cea7bdc71ced76"}
Nov 03 20:04:18.767233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:18.766792    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:18.790790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:18.790727    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4" podStartSLOduration=2.3052543979999998 podStartE2EDuration="2.79070912s" totalImagesPullingTime="485.454722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:18.787244877 +0000 UTC m=+1610.942147299" watchObservedRunningTime="2026-11-03 20:04:18.79070912 +0000 UTC m=+1610.945611538"
Nov 03 20:04:18.801232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:18.801031    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:19.591508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:19.589658    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:19.615526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:19.612499    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:19.615526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:19.614183    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:19.619506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:19.616852    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:04:19.781530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:19.777678    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4" event={"ID":"a80a4e5e-576a-4933-9e86-bb68d58cd9db","Type":"ContainerStarted","Data":"657d5ac58f7af3a0b51367917a3545642f2ae07eb0d2395bf173f393e5a7f928"}
Nov 03 20:04:20.006777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.006737    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"]
Nov 03 20:04:20.007000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.006975    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf" podUID="7280958a-874a-44e5-8360-283eeff83728" containerName="discovery" containerID="cri-o://06067807e3b04b8414242656fc57da5149a8e725a39a13948583d6d96664828e" gracePeriod=30
Nov 03 20:04:20.510819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.510792    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 20:04:20.707944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.704845    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-istio-token\" (UniqueName: \"kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-istio-token\") pod \"7280958a-874a-44e5-8360-283eeff83728\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") "
Nov 03 20:04:20.707944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.704910    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7280958a-874a-44e5-8360-283eeff83728-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7280958a-874a-44e5-8360-283eeff83728-local-certs\") pod \"7280958a-874a-44e5-8360-283eeff83728\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") "
Nov 03 20:04:20.707944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.705006    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7280958a-874a-44e5-8360-283eeff83728-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7280958a-874a-44e5-8360-283eeff83728-istio-csr-ca-configmap\") pod \"7280958a-874a-44e5-8360-283eeff83728\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") "
Nov 03 20:04:20.707944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.705054    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-cacerts\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-cacerts\") pod \"7280958a-874a-44e5-8360-283eeff83728\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") "
Nov 03 20:04:20.707944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.705130    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-kube-api-access-dsjpn\" (UniqueName: \"kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-kube-api-access-dsjpn\") pod \"7280958a-874a-44e5-8360-283eeff83728\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") "
Nov 03 20:04:20.707944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.705199    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-kubeconfig\") pod \"7280958a-874a-44e5-8360-283eeff83728\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") "
Nov 03 20:04:20.707944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.705344    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-csr-dns-cert\") pod \"7280958a-874a-44e5-8360-283eeff83728\" (UID: \"7280958a-874a-44e5-8360-283eeff83728\") "
Nov 03 20:04:20.707944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.705763    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7280958a-874a-44e5-8360-283eeff83728-istio-csr-ca-configmap" pod "7280958a-874a-44e5-8360-283eeff83728" (UID: "7280958a-874a-44e5-8360-283eeff83728"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:20.707944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.706015    5659 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7280958a-874a-44e5-8360-283eeff83728-istio-csr-ca-configmap\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:20.709599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.709507    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7280958a-874a-44e5-8360-283eeff83728-local-certs" pod "7280958a-874a-44e5-8360-283eeff83728" (UID: "7280958a-874a-44e5-8360-283eeff83728"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:20.709599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.709527    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-istio-token" pod "7280958a-874a-44e5-8360-283eeff83728" (UID: "7280958a-874a-44e5-8360-283eeff83728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:20.709599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.709562    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-csr-dns-cert" pod "7280958a-874a-44e5-8360-283eeff83728" (UID: "7280958a-874a-44e5-8360-283eeff83728"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:20.709850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.709662    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-kubeconfig" pod "7280958a-874a-44e5-8360-283eeff83728" (UID: "7280958a-874a-44e5-8360-283eeff83728"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:20.709850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.709788    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-cacerts" pod "7280958a-874a-44e5-8360-283eeff83728" (UID: "7280958a-874a-44e5-8360-283eeff83728"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:20.710898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.710874    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-kube-api-access-dsjpn" pod "7280958a-874a-44e5-8360-283eeff83728" (UID: "7280958a-874a-44e5-8360-283eeff83728"). InnerVolumeSpecName "kube-api-access-dsjpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:20.747985 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.747948    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:20.836088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.823406    5659 generic.go:356] "Generic (PLEG): container finished" podID="7280958a-874a-44e5-8360-283eeff83728" containerID="06067807e3b04b8414242656fc57da5149a8e725a39a13948583d6d96664828e" exitCode=0
Nov 03 20:04:20.836088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.824406    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"
Nov 03 20:04:20.836088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.825544    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf" event={"ID":"7280958a-874a-44e5-8360-283eeff83728","Type":"ContainerDied","Data":"06067807e3b04b8414242656fc57da5149a8e725a39a13948583d6d96664828e"}
Nov 03 20:04:20.836088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.825597    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf" event={"ID":"7280958a-874a-44e5-8360-283eeff83728","Type":"ContainerDied","Data":"e09d212107f0763f47068a655b6bda8e43f942315a6e27697402fec6cd7d1f63"}
Nov 03 20:04:20.836088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.825624    5659 scope.go:122] "RemoveContainer" containerID="06067807e3b04b8414242656fc57da5149a8e725a39a13948583d6d96664828e"
Nov 03 20:04:20.840337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.840220    5659 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-cacerts\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:20.840337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.840254    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsjpn\" (UniqueName: \"kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-kube-api-access-dsjpn\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:20.840337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.840274    5659 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-kubeconfig\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:20.840337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.840288    5659 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7280958a-874a-44e5-8360-283eeff83728-istio-csr-dns-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:20.840337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.840302    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7280958a-874a-44e5-8360-283eeff83728-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:20.840337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.840316    5659 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7280958a-874a-44e5-8360-283eeff83728-local-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:20.876351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.876329    5659 scope.go:122] "RemoveContainer" containerID="06067807e3b04b8414242656fc57da5149a8e725a39a13948583d6d96664828e"
Nov 03 20:04:20.876728 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:04:20.876707    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06067807e3b04b8414242656fc57da5149a8e725a39a13948583d6d96664828e\": container with ID starting with 06067807e3b04b8414242656fc57da5149a8e725a39a13948583d6d96664828e not found: ID does not exist" containerID="06067807e3b04b8414242656fc57da5149a8e725a39a13948583d6d96664828e"
Nov 03 20:04:20.876834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.876744    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06067807e3b04b8414242656fc57da5149a8e725a39a13948583d6d96664828e"} err="failed to get container status \"06067807e3b04b8414242656fc57da5149a8e725a39a13948583d6d96664828e\": rpc error: code = NotFound desc = could not find container \"06067807e3b04b8414242656fc57da5149a8e725a39a13948583d6d96664828e\": container with ID starting with 06067807e3b04b8414242656fc57da5149a8e725a39a13948583d6d96664828e not found: ID does not exist"
Nov 03 20:04:20.902706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.902632    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"]
Nov 03 20:04:20.914224 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:20.910501    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rcpkf"]
Nov 03 20:04:21.546344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:21.545360    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:21.970124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:21.970092    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7280958a-874a-44e5-8360-283eeff83728" path="/var/lib/kubelet/pods/7280958a-874a-44e5-8360-283eeff83728/volumes"
Nov 03 20:04:22.233670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:22.233582    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:23.153294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:23.153257    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:23.840418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:23.840365    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:24.732110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:24.732065    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:25.496509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:25.496448    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:25.831847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:25.831772    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:04:25.949009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:25.948979    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"]
Nov 03 20:04:25.949636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:25.949617    5659 cpu_manager.go:414] "containerMap: removing container" podUID="7280958a-874a-44e5-8360-283eeff83728" containerName="discovery"
Nov 03 20:04:25.949636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:25.949634    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7280958a-874a-44e5-8360-283eeff83728" containerName="discovery"
Nov 03 20:04:25.949858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:25.949844    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="7280958a-874a-44e5-8360-283eeff83728" containerName="discovery"
Nov 03 20:04:25.954800 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:25.954769    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:25.963789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:25.963657    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-f154ac\"/\"default-dockercfg-d457l\""
Nov 03 20:04:25.963789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:25.963754    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-f154ac\"/\"openshift-service-ca.crt\""
Nov 03 20:04:25.963984 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:25.963807    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-f154ac\"/\"kube-root-ca.crt\""
Nov 03 20:04:25.963984 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:25.963799    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-f154ac\"/\"auth-test-model-f154ac-kserve-self-signed-certs\""
Nov 03 20:04:25.987515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:25.986947    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"]
Nov 03 20:04:26.017918 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.017890    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:26.038598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.038566    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-dshm\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.038736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.038662    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-tmp-dir\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.038777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.038738    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-model-cache\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.038821 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.038776    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d59ee3e-419f-4f34-b4f7-095287a8f640-tls-certs\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.038934 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.038918    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-home\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.039019 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.039003    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgtt5\" (UniqueName: \"kubernetes.io/projected/5d59ee3e-419f-4f34-b4f7-095287a8f640-kube-api-access-vgtt5\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.144561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.140624    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgtt5\" (UniqueName: \"kubernetes.io/projected/5d59ee3e-419f-4f34-b4f7-095287a8f640-kube-api-access-vgtt5\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.144561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.140686    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-dshm\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.144561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.140829    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-tmp-dir\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.144561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.140945    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-model-cache\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.144561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.140986    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d59ee3e-419f-4f34-b4f7-095287a8f640-tls-certs\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.144561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.141153    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-home\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.149025 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.148982    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-tmp-dir\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.149775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.149745    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-home\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.150010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.149989    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-model-cache\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.176613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.174259    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgtt5\" (UniqueName: \"kubernetes.io/projected/5d59ee3e-419f-4f34-b4f7-095287a8f640-kube-api-access-vgtt5\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.197840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.197807    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-dshm\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.199767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.199740    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d59ee3e-419f-4f34-b4f7-095287a8f640-tls-certs\") pod \"auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") " pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.282385 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.281775    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:26.319682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.319648    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:26.659167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.659132    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"]
Nov 03 20:04:26.659823 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:04:26.659797    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d59ee3e_419f_4f34_b4f7_095287a8f640.slice/crio-bbd2b28f2360ca4bd286cf4569ffa608756b7d96fce00463f02c33b7b52f3cda WatchSource:0}: Error finding container bbd2b28f2360ca4bd286cf4569ffa608756b7d96fce00463f02c33b7b52f3cda: Status 404 returned error can't find the container with id bbd2b28f2360ca4bd286cf4569ffa608756b7d96fce00463f02c33b7b52f3cda
Nov 03 20:04:26.870748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.870706    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd" event={"ID":"5d59ee3e-419f-4f34-b4f7-095287a8f640","Type":"ContainerStarted","Data":"456ca519eadf85ef305435284d650284f37055ffb8ce30c7d3d84a974c576a33"}
Nov 03 20:04:26.871441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.870754    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd" event={"ID":"5d59ee3e-419f-4f34-b4f7-095287a8f640","Type":"ContainerStarted","Data":"bbd2b28f2360ca4bd286cf4569ffa608756b7d96fce00463f02c33b7b52f3cda"}
Nov 03 20:04:26.890478 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:26.890415    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd" podStartSLOduration=1.89039664 podStartE2EDuration="1.89039664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:26.889722797 +0000 UTC m=+1619.044625217" watchObservedRunningTime="2026-11-03 20:04:26.89039664 +0000 UTC m=+1619.045299062"
Nov 03 20:04:27.063258 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:27.063225    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:27.282954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:27.282924    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:27.852595 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:27.852560    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:28.534691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:28.534643    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:29.434055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:29.434020    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:30.329197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:30.329159    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:31.054253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:31.054216    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:31.924832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:31.924798    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:32.739864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:32.739827    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:33.449242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:33.449203    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:34.324512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:34.324450    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:34.978006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:34.977970    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:35.297702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:35.297619    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:36.282424 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.282373    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:36.296568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.296537    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:36.442057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.442023    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:36.688055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.687968    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"]
Nov 03 20:04:36.694704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.694682    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.706097 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.706069    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bfc240\"/\"kube-root-ca.crt\""
Nov 03 20:04:36.707351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.707320    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bfc240\"/\"auth-test-model-bfc240-kserve-self-signed-certs\""
Nov 03 20:04:36.707617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.707602    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bfc240\"/\"default-dockercfg-h25rf\""
Nov 03 20:04:36.708952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.707704    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bfc240\"/\"openshift-service-ca.crt\""
Nov 03 20:04:36.713230 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.713200    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"]
Nov 03 20:04:36.724019 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.723994    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-model-cache\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.724344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.724322    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tmp-dir\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.724581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.724545    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7ldn\" (UniqueName: \"kubernetes.io/projected/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-kube-api-access-g7ldn\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.726371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.724760    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tls-certs\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.726646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.726626    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-home\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.727728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.727703    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-dshm\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.816734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.816697    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:36.829573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.829536    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-dshm\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.829767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.829675    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-model-cache\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.829767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.829762    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tmp-dir\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.829892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.829808    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7ldn\" (UniqueName: \"kubernetes.io/projected/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-kube-api-access-g7ldn\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.829892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.829850    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tls-certs\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.829997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.829895    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-home\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.830266 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.830229    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tmp-dir\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.830399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.830381    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-model-cache\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.830633 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.830610    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-home\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.832345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.832319    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-dshm\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.833044 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.833016    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tls-certs\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:36.843005 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:36.842981    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7ldn\" (UniqueName: \"kubernetes.io/projected/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-kube-api-access-g7ldn\") pod \"auth-test-model-bfc240-kserve-6987dbf9b9-247h5\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") " pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:37.015413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:37.015374    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:37.030939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:37.030902    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:37.148001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:37.147971    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:04:37.425572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:37.425545    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"]
Nov 03 20:04:37.426580 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:04:37.426544    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81b2ae07_b323_4ad4_99fc_45d5ad248aa0.slice/crio-267591add7376b55cf0b22eec38c52804887c67dcb7967bfe0afdbc440e23db2 WatchSource:0}: Error finding container 267591add7376b55cf0b22eec38c52804887c67dcb7967bfe0afdbc440e23db2: Status 404 returned error can't find the container with id 267591add7376b55cf0b22eec38c52804887c67dcb7967bfe0afdbc440e23db2
Nov 03 20:04:37.435907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:37.435881    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:37.995611 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:37.995574    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5" event={"ID":"81b2ae07-b323-4ad4-99fc-45d5ad248aa0","Type":"ContainerStarted","Data":"0d0f879d32e986e8c9a16cf250d7634df1362c5008ee2c6c26995b774069d1ad"}
Nov 03 20:04:37.995801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:37.995624    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5" event={"ID":"81b2ae07-b323-4ad4-99fc-45d5ad248aa0","Type":"ContainerStarted","Data":"267591add7376b55cf0b22eec38c52804887c67dcb7967bfe0afdbc440e23db2"}
Nov 03 20:04:38.017118 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:38.017071    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5" podStartSLOduration=2.017055788 podStartE2EDuration="2.017055788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:38.015126931 +0000 UTC m=+1630.170029351" watchObservedRunningTime="2026-11-03 20:04:38.017055788 +0000 UTC m=+1630.171958206"
Nov 03 20:04:38.031401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:38.031373    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:38.094036 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:38.094005    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:38.942530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:38.942438    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:39.846834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:39.846404    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:40.593014 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:40.592975    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:41.249334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:41.249299    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:42.015930 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:42.015886    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:42.776193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:42.776145    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:43.460559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:43.460527    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:44.151399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:44.151268    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:44.841757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:44.841717    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:45.219521 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:04:45.219465    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:04:45.525626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.525544    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:45.728334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.728295    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"]
Nov 03 20:04:45.734622 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.734597    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:45.743517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.742017    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-78d47a\"/\"default-dockercfg-4qr2c\""
Nov 03 20:04:45.743517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.742090    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-78d47a\"/\"kube-root-ca.crt\""
Nov 03 20:04:45.743517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.742091    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-78d47a\"/\"explicit-model-78d47a-kserve-self-signed-certs\""
Nov 03 20:04:45.743517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.742091    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-78d47a\"/\"openshift-service-ca.crt\""
Nov 03 20:04:45.743517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.742598    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"]
Nov 03 20:04:45.810555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.810331    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:45.898942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.898905    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-model-cache\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:45.899446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.898968    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-tmp-dir\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:45.899446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.899038    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-dshm\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:45.899446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.899224    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d69463a1-de88-46e7-a692-7c0b45161c5d-tls-certs\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:45.899446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.899300    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dk8h\" (UniqueName: \"kubernetes.io/projected/d69463a1-de88-46e7-a692-7c0b45161c5d-kube-api-access-4dk8h\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:45.899446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:45.899398    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-home\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.010506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.006029    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d69463a1-de88-46e7-a692-7c0b45161c5d-tls-certs\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.035507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.014840    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dk8h\" (UniqueName: \"kubernetes.io/projected/d69463a1-de88-46e7-a692-7c0b45161c5d-kube-api-access-4dk8h\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.035507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.014985    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-home\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.035507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.015418    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-home\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.035507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.015658    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-model-cache\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.035507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.015948    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-model-cache\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.035507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.016020    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-tmp-dir\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.035507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.016269    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-tmp-dir\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.035507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.016380    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-dshm\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.035507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.035237    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dk8h\" (UniqueName: \"kubernetes.io/projected/d69463a1-de88-46e7-a692-7c0b45161c5d-kube-api-access-4dk8h\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.044348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.044287    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d69463a1-de88-46e7-a692-7c0b45161c5d-tls-certs\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.047350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.047277    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-dshm\") pod \"explicit-model-78d47a-kserve-6f64cff457-925bg\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") " pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.067711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.066796    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:46.403282 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.397641    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"]
Nov 03 20:04:46.428668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:46.428632    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:47.032366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:47.032337    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:47.045527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:47.045500    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:47.075976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:47.075929    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg" event={"ID":"d69463a1-de88-46e7-a692-7c0b45161c5d","Type":"ContainerStarted","Data":"4b942767406c5a135aaff9a596a86644ad4c68fc3785a500f920d667fbc3bac7"}
Nov 03 20:04:47.075976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:47.075971    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg" event={"ID":"d69463a1-de88-46e7-a692-7c0b45161c5d","Type":"ContainerStarted","Data":"a19fa9ef8a2f173efda87e22c8a43550115fdadfa596dd97f11f3dc3e96182af"}
Nov 03 20:04:47.087812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:47.087784    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:04:47.100621 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:47.100571    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg" podStartSLOduration=2.100555255 podStartE2EDuration="2.100555255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:47.099618194 +0000 UTC m=+1639.254520625" watchObservedRunningTime="2026-11-03 20:04:47.100555255 +0000 UTC m=+1639.255457674"
Nov 03 20:04:47.205870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:47.205823    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:47.963365 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:47.963278    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:48.067439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:48.067401    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:04:48.630831 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:48.630787    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:49.296079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:49.296028    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:49.439506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:49.432947    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:50.059803 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:50.059762    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:50.262305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:50.262273    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:50.953524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:50.953423    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:51.682255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:51.682225    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:52.162725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:52.162687    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:52.287774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:52.287743    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:52.434906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:52.434801    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:53.156364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:53.156330    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:53.518212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:53.518178    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"]
Nov 03 20:04:53.644188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:53.644134    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:54.114219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.114182    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:54.267676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.267645    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:54.382073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.381993    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"]
Nov 03 20:04:54.388746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.388709    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"]
Nov 03 20:04:54.388899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.388827    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.394914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.394386    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-43c674\"/\"default-dockercfg-jtdmn\""
Nov 03 20:04:54.399360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.395552    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-43c674\"/\"openshift-service-ca.crt\""
Nov 03 20:04:54.399360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.395638    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-43c674\"/\"auth-test-model-43c674-kserve-self-signed-certs\""
Nov 03 20:04:54.399360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.395779    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-43c674\"/\"kube-root-ca.crt\""
Nov 03 20:04:54.425375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.425350    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:54.495083 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.495041    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-tmp-dir\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.495289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.495258    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-model-cache\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.495289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.495281    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-dshm\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.495417 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.495300    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-home\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.495417 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.495327    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnsvz\" (UniqueName: \"kubernetes.io/projected/1c26616c-2785-4639-8c41-73b452fe827f-kube-api-access-fnsvz\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.495586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.495558    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c26616c-2785-4639-8c41-73b452fe827f-tls-certs\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.602506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.599391    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-model-cache\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.602506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.599452    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-dshm\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.602506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.599499    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-home\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.602506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.599528    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnsvz\" (UniqueName: \"kubernetes.io/projected/1c26616c-2785-4639-8c41-73b452fe827f-kube-api-access-fnsvz\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.602506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.599623    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c26616c-2785-4639-8c41-73b452fe827f-tls-certs\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.602506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.599685    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-tmp-dir\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.602506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.600345    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-tmp-dir\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.602506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.600612    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-model-cache\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.602506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.601850    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-home\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.608970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.606794    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-dshm\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.626721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.613084    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c26616c-2785-4639-8c41-73b452fe827f-tls-certs\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.626721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.615197    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnsvz\" (UniqueName: \"kubernetes.io/projected/1c26616c-2785-4639-8c41-73b452fe827f-kube-api-access-fnsvz\") pod \"auth-test-model-43c674-kserve-9477cf84f-pf8gv\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") " pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:54.725379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:54.725349    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:55.040251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:55.040221    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"]
Nov 03 20:04:55.042518 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:04:55.042472    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c26616c_2785_4639_8c41_73b452fe827f.slice/crio-7448835309ec8c6a2843c277fbc019d0c4ced257e9efa7d370bf88297acfc2e0 WatchSource:0}: Error finding container 7448835309ec8c6a2843c277fbc019d0c4ced257e9efa7d370bf88297acfc2e0: Status 404 returned error can't find the container with id 7448835309ec8c6a2843c277fbc019d0c4ced257e9efa7d370bf88297acfc2e0
Nov 03 20:04:55.155707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:55.155664    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv" event={"ID":"1c26616c-2785-4639-8c41-73b452fe827f","Type":"ContainerStarted","Data":"0a2f41786727f2a67b3e4697ab403cf19716b5de4c0c3d876af155de65330969"}
Nov 03 20:04:55.155878 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:55.155724    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv" event={"ID":"1c26616c-2785-4639-8c41-73b452fe827f","Type":"ContainerStarted","Data":"7448835309ec8c6a2843c277fbc019d0c4ced257e9efa7d370bf88297acfc2e0"}
Nov 03 20:04:55.181509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:55.178237    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv" podStartSLOduration=1.178189342 podStartE2EDuration="1.178189342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:55.175860493 +0000 UTC m=+1647.330762937" watchObservedRunningTime="2026-11-03 20:04:55.178189342 +0000 UTC m=+1647.333091813"
Nov 03 20:04:55.182338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:55.182303    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:55.726165 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:55.726126    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:04:55.842841 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:55.842809    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:56.532178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:56.532146    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:57.228063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:57.228026    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:57.932536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:57.932499    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:58.620534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:58.620472    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:59.071826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:59.071793    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:59.362953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:04:59.362871    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:00.026365 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:00.026334    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:00.867034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:00.866998    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:01.652108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:01.652046    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:02.730619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:02.730584    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:03.914268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:03.914223    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:04.725901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:04.725869    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:05:04.743777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:04.743741    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:05:04.748511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:04.747178    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:05.284501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:05.284459    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:05:05.587577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:05.587420    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:06.337740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:06.337706    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:07.133554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:07.133520    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:07.827697 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:07.827662    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:08.545630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.545585    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg" podUID="d69463a1-de88-46e7-a692-7c0b45161c5d" containerName="main" containerID="cri-o://4b942767406c5a135aaff9a596a86644ad4c68fc3785a500f920d667fbc3bac7" gracePeriod=45
Nov 03 20:05:08.655650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.654959    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:08.861081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.861058    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:05:08.949593 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.949556    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d69463a1-de88-46e7-a692-7c0b45161c5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d69463a1-de88-46e7-a692-7c0b45161c5d-tls-certs\") pod \"d69463a1-de88-46e7-a692-7c0b45161c5d\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") "
Nov 03 20:05:08.949773 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.949736    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-dshm\") pod \"d69463a1-de88-46e7-a692-7c0b45161c5d\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") "
Nov 03 20:05:08.949901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.949861    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-tmp-dir\") pod \"d69463a1-de88-46e7-a692-7c0b45161c5d\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") "
Nov 03 20:05:08.950049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.949913    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-home\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-home\") pod \"d69463a1-de88-46e7-a692-7c0b45161c5d\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") "
Nov 03 20:05:08.950049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.950005    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69463a1-de88-46e7-a692-7c0b45161c5d-kube-api-access-4dk8h\" (UniqueName: \"kubernetes.io/projected/d69463a1-de88-46e7-a692-7c0b45161c5d-kube-api-access-4dk8h\") pod \"d69463a1-de88-46e7-a692-7c0b45161c5d\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") "
Nov 03 20:05:08.950158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.950048    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-model-cache\") pod \"d69463a1-de88-46e7-a692-7c0b45161c5d\" (UID: \"d69463a1-de88-46e7-a692-7c0b45161c5d\") "
Nov 03 20:05:08.950158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.950089    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-tmp-dir" pod "d69463a1-de88-46e7-a692-7c0b45161c5d" (UID: "d69463a1-de88-46e7-a692-7c0b45161c5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:08.950323 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.950285    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-home" pod "d69463a1-de88-46e7-a692-7c0b45161c5d" (UID: "d69463a1-de88-46e7-a692-7c0b45161c5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:08.950608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.950581    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-model-cache" pod "d69463a1-de88-46e7-a692-7c0b45161c5d" (UID: "d69463a1-de88-46e7-a692-7c0b45161c5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:08.950890 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.950871    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:08.950982 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.950896    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:08.950982 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.950913    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:08.952457 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.952429    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d69463a1-de88-46e7-a692-7c0b45161c5d-tls-certs" pod "d69463a1-de88-46e7-a692-7c0b45161c5d" (UID: "d69463a1-de88-46e7-a692-7c0b45161c5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:08.952573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.952524    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-dshm" pod "d69463a1-de88-46e7-a692-7c0b45161c5d" (UID: "d69463a1-de88-46e7-a692-7c0b45161c5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:08.952648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:08.952619    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69463a1-de88-46e7-a692-7c0b45161c5d-kube-api-access-4dk8h" pod "d69463a1-de88-46e7-a692-7c0b45161c5d" (UID: "d69463a1-de88-46e7-a692-7c0b45161c5d"). InnerVolumeSpecName "kube-api-access-4dk8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:09.052178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.052137    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d69463a1-de88-46e7-a692-7c0b45161c5d-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:09.052178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.052176    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dk8h\" (UniqueName: \"kubernetes.io/projected/d69463a1-de88-46e7-a692-7c0b45161c5d-kube-api-access-4dk8h\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:09.052405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.052194    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d69463a1-de88-46e7-a692-7c0b45161c5d-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:09.303740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.303699    5659 generic.go:356] "Generic (PLEG): container finished" podID="d69463a1-de88-46e7-a692-7c0b45161c5d" containerID="4b942767406c5a135aaff9a596a86644ad4c68fc3785a500f920d667fbc3bac7" exitCode=0
Nov 03 20:05:09.303955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.303747    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"
Nov 03 20:05:09.303955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.303777    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg" event={"ID":"d69463a1-de88-46e7-a692-7c0b45161c5d","Type":"ContainerDied","Data":"4b942767406c5a135aaff9a596a86644ad4c68fc3785a500f920d667fbc3bac7"}
Nov 03 20:05:09.303955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.303822    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg" event={"ID":"d69463a1-de88-46e7-a692-7c0b45161c5d","Type":"ContainerDied","Data":"a19fa9ef8a2f173efda87e22c8a43550115fdadfa596dd97f11f3dc3e96182af"}
Nov 03 20:05:09.303955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.303840    5659 scope.go:122] "RemoveContainer" containerID="4b942767406c5a135aaff9a596a86644ad4c68fc3785a500f920d667fbc3bac7"
Nov 03 20:05:09.314893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.314874    5659 scope.go:122] "RemoveContainer" containerID="4b942767406c5a135aaff9a596a86644ad4c68fc3785a500f920d667fbc3bac7"
Nov 03 20:05:09.315185 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:05:09.315165    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b942767406c5a135aaff9a596a86644ad4c68fc3785a500f920d667fbc3bac7\": container with ID starting with 4b942767406c5a135aaff9a596a86644ad4c68fc3785a500f920d667fbc3bac7 not found: ID does not exist" containerID="4b942767406c5a135aaff9a596a86644ad4c68fc3785a500f920d667fbc3bac7"
Nov 03 20:05:09.315233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.315194    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b942767406c5a135aaff9a596a86644ad4c68fc3785a500f920d667fbc3bac7"} err="failed to get container status \"4b942767406c5a135aaff9a596a86644ad4c68fc3785a500f920d667fbc3bac7\": rpc error: code = NotFound desc = could not find container \"4b942767406c5a135aaff9a596a86644ad4c68fc3785a500f920d667fbc3bac7\": container with ID starting with 4b942767406c5a135aaff9a596a86644ad4c68fc3785a500f920d667fbc3bac7 not found: ID does not exist"
Nov 03 20:05:09.328184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.328158    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"]
Nov 03 20:05:09.334436 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.334407    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-78d47a/explicit-model-78d47a-kserve-6f64cff457-925bg"]
Nov 03 20:05:09.356242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.356212    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:09.971817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:09.971781    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69463a1-de88-46e7-a692-7c0b45161c5d" path="/var/lib/kubelet/pods/d69463a1-de88-46e7-a692-7c0b45161c5d/volumes"
Nov 03 20:05:10.267979 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:10.267882    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:10.963999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:10.963967    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:11.639253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:11.639214    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:11.946812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:11.946691    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:12.541338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:12.541304    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:13.333647 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:13.333612    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:14.144528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:14.139815    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:14.830045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:14.830017    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:15.217904 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:05:15.217865    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:05:15.567474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:15.567372    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:16.348838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:16.348808    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:17.056852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:17.056820    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:17.938588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:17.938556    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:18.738112 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:18.738081    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:19.427102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:19.427072    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:20.140184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:20.140145    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:20.849175 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:20.849109    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:21.660898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:21.660862    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:22.394504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:22.392966    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:23.044248 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:23.044209    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:23.771061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:23.771025    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:24.429069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:24.429040    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:25.066421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:25.066373    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:25.728601 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:25.728564    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:26.351383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:26.351347    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:27.083932 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.083902    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:27.763976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.763939    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:27.791708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.791660    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"]
Nov 03 20:05:27.792599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.792549    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d69463a1-de88-46e7-a692-7c0b45161c5d" containerName="main"
Nov 03 20:05:27.792599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.792595    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69463a1-de88-46e7-a692-7c0b45161c5d" containerName="main"
Nov 03 20:05:27.792875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.792858    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69463a1-de88-46e7-a692-7c0b45161c5d" containerName="main"
Nov 03 20:05:27.797655 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.797627    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"]
Nov 03 20:05:27.797879 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.797860    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.806015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.805979    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-81a5ae-openshift-default-dockercfg-pvl2f\""
Nov 03 20:05:27.888633 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.888600    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fe4f76-2186-4315-bb74-db93367de753-istiod-ca-cert\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.888840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.888731    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-certs\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.888927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.888841    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-credential-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.888927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.888895    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-envoy\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.889017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.888975    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-data\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.889017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.889001    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-istio-token\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.889111 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.889038    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.889172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.889157    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fe4f76-2186-4315-bb74-db93367de753-istio-podinfo\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.889234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.889218    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d672n\" (UniqueName: \"kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-kube-api-access-d672n\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.989900 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.989866    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-certs\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.990073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.989912    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-credential-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.990073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.989942    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-envoy\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.990073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.989960    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-data\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.990073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.989976    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-istio-token\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.990073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.990043    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.990344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.990169    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fe4f76-2186-4315-bb74-db93367de753-istio-podinfo\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.990344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.990210    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d672n\" (UniqueName: \"kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-kube-api-access-d672n\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.990344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.990295    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-certs\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.990543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.990368    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-credential-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.990543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.990395    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fe4f76-2186-4315-bb74-db93367de753-istiod-ca-cert\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.990543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.990431    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-data\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.991032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.990808    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-socket\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.991349 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.991328    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fe4f76-2186-4315-bb74-db93367de753-istiod-ca-cert\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.992803 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.992779    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fe4f76-2186-4315-bb74-db93367de753-istio-podinfo\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:27.992963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:27.992945    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-envoy\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:28.003044 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.003023    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d672n\" (UniqueName: \"kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-kube-api-access-d672n\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:28.003144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.003081    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-istio-token\") pod \"e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") " pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:28.124087 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.124014    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-81a5ae-openshift-default-dockercfg-pvl2f\""
Nov 03 20:05:28.132126 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.132091    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:28.384450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.384370    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"]
Nov 03 20:05:28.389878 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.389790    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.394675 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.394653    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-f154ac-openshift-default-dockercfg-xndgl\""
Nov 03 20:05:28.395368 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.395339    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-data\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.395506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.395393    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4922de3-227d-479c-a91d-1db7e320b89b-istio-podinfo\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.395506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.395444    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-envoy\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.395627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.395555    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5kkg\" (UniqueName: \"kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-kube-api-access-w5kkg\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.395778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.395761    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-credential-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.395858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.395800    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-istio-token\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.395858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.395848    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.395972 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.395876    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-certs\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.395972 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.395906    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4922de3-227d-479c-a91d-1db7e320b89b-istiod-ca-cert\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.400752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.400717    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"]
Nov 03 20:05:28.420757 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:05:28.420617    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2fe4f76_2186_4315_bb74_db93367de753.slice/crio-e579dabca1ac13a5fae6134bfa38f5e522e9eac3f8b76b2f4fb85840bfbc9322 WatchSource:0}: Error finding container e579dabca1ac13a5fae6134bfa38f5e522e9eac3f8b76b2f4fb85840bfbc9322: Status 404 returned error can't find the container with id e579dabca1ac13a5fae6134bfa38f5e522e9eac3f8b76b2f4fb85840bfbc9322
Nov 03 20:05:28.423080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.423042    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:28.423198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.423112    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:28.423198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.423146    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:28.426518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.426496    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"]
Nov 03 20:05:28.496750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.496726    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-envoy\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.496848 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.496811    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5kkg\" (UniqueName: \"kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-kube-api-access-w5kkg\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.497006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.496991    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-credential-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.497076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.497029    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-istio-token\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.497139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.497074    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.497139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.497106    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-certs\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.497243 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.497139    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4922de3-227d-479c-a91d-1db7e320b89b-istiod-ca-cert\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.497243 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.497204    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-data\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.497352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.497239    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4922de3-227d-479c-a91d-1db7e320b89b-istio-podinfo\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.499110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.498732    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-credential-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.499110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.498781    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-certs\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.499110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.499032    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.499314 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.499140    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-data\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.499314 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.499163    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4922de3-227d-479c-a91d-1db7e320b89b-istiod-ca-cert\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.499975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.499957    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-envoy\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.500104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.500087    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4922de3-227d-479c-a91d-1db7e320b89b-istio-podinfo\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.507156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.507126    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" event={"ID":"b2fe4f76-2186-4315-bb74-db93367de753","Type":"ContainerStarted","Data":"0b8e0b92a00848ffd1d93702dfe8474f28204a3e5db8cdb5baeadc767a7d946a"}
Nov 03 20:05:28.507534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.507190    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" event={"ID":"b2fe4f76-2186-4315-bb74-db93367de753","Type":"ContainerStarted","Data":"e579dabca1ac13a5fae6134bfa38f5e522e9eac3f8b76b2f4fb85840bfbc9322"}
Nov 03 20:05:28.509373 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.509350    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-istio-token\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.510521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.510468    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5kkg\" (UniqueName: \"kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-kube-api-access-w5kkg\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:28.529926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.529852    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" podStartSLOduration=1.529838631 podStartE2EDuration="1.529838631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:28.52946297 +0000 UTC m=+1680.684365412" watchObservedRunningTime="2026-11-03 20:05:28.529838631 +0000 UTC m=+1680.684741051"
Nov 03 20:05:28.545158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.545087    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:28.709251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:28.709228    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:29.004079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:29.004050    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"]
Nov 03 20:05:29.004542 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:05:29.004516    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4922de3_227d_479c_a91d_1db7e320b89b.slice/crio-8a81e389b1e70c8bcc4684f5c0bb9f30f88370c1339b07c7e610f563170bea00 WatchSource:0}: Error finding container 8a81e389b1e70c8bcc4684f5c0bb9f30f88370c1339b07c7e610f563170bea00: Status 404 returned error can't find the container with id 8a81e389b1e70c8bcc4684f5c0bb9f30f88370c1339b07c7e610f563170bea00
Nov 03 20:05:29.007769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:29.007734    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:29.007875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:29.007797    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:29.007875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:29.007827    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:29.133190 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:29.133158    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:29.134598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:29.134570    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 03 20:05:29.134729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:29.134632    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" podUID="b2fe4f76-2186-4315-bb74-db93367de753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 03 20:05:29.315859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:29.315762    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:29.516240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:29.516209    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn" event={"ID":"d4922de3-227d-479c-a91d-1db7e320b89b","Type":"ContainerStarted","Data":"eb646c6e8dfff224473fd85b5bffe8dde9be63216bef7649791dad8001c7783b"}
Nov 03 20:05:29.516425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:29.516259    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn" event={"ID":"d4922de3-227d-479c-a91d-1db7e320b89b","Type":"ContainerStarted","Data":"8a81e389b1e70c8bcc4684f5c0bb9f30f88370c1339b07c7e610f563170bea00"}
Nov 03 20:05:29.710357 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:29.710332    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:29.949107 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:29.948915    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:30.135949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:30.135908    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 03 20:05:30.136581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:30.136538    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" podUID="b2fe4f76-2186-4315-bb74-db93367de753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 03 20:05:30.710635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:30.710593    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:05:30.710843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:30.710647    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn" podUID="d4922de3-227d-479c-a91d-1db7e320b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 20:05:30.778515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:30.778446    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:31.152563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:31.140965    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 03 20:05:31.152563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:31.141029    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" podUID="b2fe4f76-2186-4315-bb74-db93367de753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 03 20:05:31.567506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:31.565360    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:31.721507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:31.712930    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:05:31.721507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:31.713008    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn" podUID="d4922de3-227d-479c-a91d-1db7e320b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 20:05:32.143506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:32.139627    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 03 20:05:32.143506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:32.139688    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" podUID="b2fe4f76-2186-4315-bb74-db93367de753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 03 20:05:32.463507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:32.463181    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:32.710505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:32.709919    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:05:32.710505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:32.709980    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn" podUID="d4922de3-227d-479c-a91d-1db7e320b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 20:05:33.138501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:33.136107    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 03 20:05:33.138501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:33.136167    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" podUID="b2fe4f76-2186-4315-bb74-db93367de753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 03 20:05:33.511705 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:33.511665    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:33.710513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:33.710468    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:05:33.710748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:33.710727    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn" podUID="d4922de3-227d-479c-a91d-1db7e320b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 20:05:34.147931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:34.147895    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 03 20:05:34.148242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:34.148205    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" podUID="b2fe4f76-2186-4315-bb74-db93367de753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 03 20:05:34.255274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:34.255242    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:34.710193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:34.710157    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:05:34.710819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:34.710772    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn" podUID="d4922de3-227d-479c-a91d-1db7e320b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 20:05:34.954147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:34.954107    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:35.156171 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:35.156078    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 03 20:05:35.156446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:35.156413    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" podUID="b2fe4f76-2186-4315-bb74-db93367de753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 03 20:05:35.398798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:35.398740    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn" podStartSLOduration=7.398721542 podStartE2EDuration="7.398721542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:29.552531221 +0000 UTC m=+1681.707433685" watchObservedRunningTime="2026-11-03 20:05:35.398721542 +0000 UTC m=+1687.553623962"
Nov 03 20:05:35.404128 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:35.404102    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"]
Nov 03 20:05:35.419404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:35.419315    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"]
Nov 03 20:05:35.519070 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:35.518938    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:35.651507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:35.649818    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:35.728508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:35.720899    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:05:35.728508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:35.720962    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn" podUID="d4922de3-227d-479c-a91d-1db7e320b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 20:05:36.067943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:36.067849    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:36.176008 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:36.175971    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:36.176580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:36.176476    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:36.177774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:36.177686    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:05:36.766301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:36.766271    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:36.766863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:36.766737    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:36.768193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:36.767635    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:05:36.870876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:36.870844    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:37.012370 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:37.012328    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:37.145221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:37.145105    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:37.729861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:37.729820    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:38.524343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:38.524306    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:39.285196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:39.285167    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:39.781433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:39.781395    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"]
Nov 03 20:05:39.782064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:39.781664    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4" podUID="a80a4e5e-576a-4933-9e86-bb68d58cd9db" containerName="maas-api" containerID="cri-o://657d5ac58f7af3a0b51367917a3545642f2ae07eb0d2395bf173f393e5a7f928" gracePeriod=30
Nov 03 20:05:39.857343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:39.857309    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:40.165560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.163282    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:40.371122 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.371080    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:05:40.404579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.404538    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf" podUID="3f623bbd-bc79-4756-b7fb-f4d656ebddff" containerName="payload-processing" containerID="cri-o://ab79d9804869c9cd74535af043c1601ab08642e5972f2b7b5ef50ba21bf66434" gracePeriod=25
Nov 03 20:05:40.420534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.420442    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj" podUID="49286968-069b-4864-a7bc-aedbb7d8a258" containerName="payload-pre-processing" containerID="cri-o://8e154b611c9aa565536d5612f4efdff490b5ac936639a72999f7726f1fa87789" gracePeriod=25
Nov 03 20:05:40.548505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.537851    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-metrics-certs\") pod \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") "
Nov 03 20:05:40.548505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.538060    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80a4e5e-576a-4933-9e86-bb68d58cd9db-kube-api-access-hpkh9\" (UniqueName: \"kubernetes.io/projected/a80a4e5e-576a-4933-9e86-bb68d58cd9db-kube-api-access-hpkh9\") pod \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") "
Nov 03 20:05:40.548505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.538127    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-maas-api-tls\") pod \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") "
Nov 03 20:05:40.548505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.538157    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80a4e5e-576a-4933-9e86-bb68d58cd9db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80a4e5e-576a-4933-9e86-bb68d58cd9db-openshift-service-ca\") pod \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\" (UID: \"a80a4e5e-576a-4933-9e86-bb68d58cd9db\") "
Nov 03 20:05:40.548505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.538976    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80a4e5e-576a-4933-9e86-bb68d58cd9db-openshift-service-ca" pod "a80a4e5e-576a-4933-9e86-bb68d58cd9db" (UID: "a80a4e5e-576a-4933-9e86-bb68d58cd9db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:40.548505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.542243    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-maas-api-tls" pod "a80a4e5e-576a-4933-9e86-bb68d58cd9db" (UID: "a80a4e5e-576a-4933-9e86-bb68d58cd9db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:40.555527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.550927    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-metrics-certs" pod "a80a4e5e-576a-4933-9e86-bb68d58cd9db" (UID: "a80a4e5e-576a-4933-9e86-bb68d58cd9db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:40.555527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.550979    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80a4e5e-576a-4933-9e86-bb68d58cd9db-kube-api-access-hpkh9" pod "a80a4e5e-576a-4933-9e86-bb68d58cd9db" (UID: "a80a4e5e-576a-4933-9e86-bb68d58cd9db"). InnerVolumeSpecName "kube-api-access-hpkh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:40.640337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.640297    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:40.640337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.640332    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpkh9\" (UniqueName: \"kubernetes.io/projected/a80a4e5e-576a-4933-9e86-bb68d58cd9db-kube-api-access-hpkh9\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:40.640552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.640348    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80a4e5e-576a-4933-9e86-bb68d58cd9db-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:40.640552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.640364    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80a4e5e-576a-4933-9e86-bb68d58cd9db-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:40.662233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.662199    5659 generic.go:356] "Generic (PLEG): container finished" podID="49286968-069b-4864-a7bc-aedbb7d8a258" containerID="8e154b611c9aa565536d5612f4efdff490b5ac936639a72999f7726f1fa87789" exitCode=0
Nov 03 20:05:40.662431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.662274    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj" event={"ID":"49286968-069b-4864-a7bc-aedbb7d8a258","Type":"ContainerDied","Data":"8e154b611c9aa565536d5612f4efdff490b5ac936639a72999f7726f1fa87789"}
Nov 03 20:05:40.663990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.663849    5659 generic.go:356] "Generic (PLEG): container finished" podID="a80a4e5e-576a-4933-9e86-bb68d58cd9db" containerID="657d5ac58f7af3a0b51367917a3545642f2ae07eb0d2395bf173f393e5a7f928" exitCode=0
Nov 03 20:05:40.663990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.663926    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"
Nov 03 20:05:40.663990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.663958    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4" event={"ID":"a80a4e5e-576a-4933-9e86-bb68d58cd9db","Type":"ContainerDied","Data":"657d5ac58f7af3a0b51367917a3545642f2ae07eb0d2395bf173f393e5a7f928"}
Nov 03 20:05:40.664209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.664193    5659 scope.go:122] "RemoveContainer" containerID="657d5ac58f7af3a0b51367917a3545642f2ae07eb0d2395bf173f393e5a7f928"
Nov 03 20:05:40.665310 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.665227    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4" event={"ID":"a80a4e5e-576a-4933-9e86-bb68d58cd9db","Type":"ContainerDied","Data":"287c8ec8b8260a00da729884c8af6143fad9cd7cb01be5efb9cea7bdc71ced76"}
Nov 03 20:05:40.667034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.667008    5659 generic.go:356] "Generic (PLEG): container finished" podID="3f623bbd-bc79-4756-b7fb-f4d656ebddff" containerID="ab79d9804869c9cd74535af043c1601ab08642e5972f2b7b5ef50ba21bf66434" exitCode=0
Nov 03 20:05:40.667151 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.667113    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf" event={"ID":"3f623bbd-bc79-4756-b7fb-f4d656ebddff","Type":"ContainerDied","Data":"ab79d9804869c9cd74535af043c1601ab08642e5972f2b7b5ef50ba21bf66434"}
Nov 03 20:05:40.679432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.679409    5659 scope.go:122] "RemoveContainer" containerID="657d5ac58f7af3a0b51367917a3545642f2ae07eb0d2395bf173f393e5a7f928"
Nov 03 20:05:40.679790 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:05:40.679762    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657d5ac58f7af3a0b51367917a3545642f2ae07eb0d2395bf173f393e5a7f928\": container with ID starting with 657d5ac58f7af3a0b51367917a3545642f2ae07eb0d2395bf173f393e5a7f928 not found: ID does not exist" containerID="657d5ac58f7af3a0b51367917a3545642f2ae07eb0d2395bf173f393e5a7f928"
Nov 03 20:05:40.679873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.679801    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657d5ac58f7af3a0b51367917a3545642f2ae07eb0d2395bf173f393e5a7f928"} err="failed to get container status \"657d5ac58f7af3a0b51367917a3545642f2ae07eb0d2395bf173f393e5a7f928\": rpc error: code = NotFound desc = could not find container \"657d5ac58f7af3a0b51367917a3545642f2ae07eb0d2395bf173f393e5a7f928\": container with ID starting with 657d5ac58f7af3a0b51367917a3545642f2ae07eb0d2395bf173f393e5a7f928 not found: ID does not exist"
Nov 03 20:05:40.689786 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.689768    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"
Nov 03 20:05:40.707228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.707208    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"]
Nov 03 20:05:40.723975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.723948    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"
Nov 03 20:05:40.725244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.725223    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-78d47a-69bbdfdccf-7z6p4"]
Nov 03 20:05:40.842363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.842329    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f623bbd-bc79-4756-b7fb-f4d656ebddff-kube-api-access-68m56\" (UniqueName: \"kubernetes.io/projected/3f623bbd-bc79-4756-b7fb-f4d656ebddff-kube-api-access-68m56\") pod \"3f623bbd-bc79-4756-b7fb-f4d656ebddff\" (UID: \"3f623bbd-bc79-4756-b7fb-f4d656ebddff\") "
Nov 03 20:05:40.842788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.842406    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49286968-069b-4864-a7bc-aedbb7d8a258-kube-api-access-zn8n8\" (UniqueName: \"kubernetes.io/projected/49286968-069b-4864-a7bc-aedbb7d8a258-kube-api-access-zn8n8\") pod \"49286968-069b-4864-a7bc-aedbb7d8a258\" (UID: \"49286968-069b-4864-a7bc-aedbb7d8a258\") "
Nov 03 20:05:40.842788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.842458    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f623bbd-bc79-4756-b7fb-f4d656ebddff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f623bbd-bc79-4756-b7fb-f4d656ebddff-plugins-config-volume\") pod \"3f623bbd-bc79-4756-b7fb-f4d656ebddff\" (UID: \"3f623bbd-bc79-4756-b7fb-f4d656ebddff\") "
Nov 03 20:05:40.842788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.842519    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49286968-069b-4864-a7bc-aedbb7d8a258-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49286968-069b-4864-a7bc-aedbb7d8a258-plugins-config-volume\") pod \"49286968-069b-4864-a7bc-aedbb7d8a258\" (UID: \"49286968-069b-4864-a7bc-aedbb7d8a258\") "
Nov 03 20:05:40.842954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.842885    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49286968-069b-4864-a7bc-aedbb7d8a258-plugins-config-volume" pod "49286968-069b-4864-a7bc-aedbb7d8a258" (UID: "49286968-069b-4864-a7bc-aedbb7d8a258"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:40.842954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.842894    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f623bbd-bc79-4756-b7fb-f4d656ebddff-plugins-config-volume" pod "3f623bbd-bc79-4756-b7fb-f4d656ebddff" (UID: "3f623bbd-bc79-4756-b7fb-f4d656ebddff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:40.843131 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.843115    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f623bbd-bc79-4756-b7fb-f4d656ebddff-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:40.843169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.843135    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49286968-069b-4864-a7bc-aedbb7d8a258-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:40.844513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.844469    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f623bbd-bc79-4756-b7fb-f4d656ebddff-kube-api-access-68m56" pod "3f623bbd-bc79-4756-b7fb-f4d656ebddff" (UID: "3f623bbd-bc79-4756-b7fb-f4d656ebddff"). InnerVolumeSpecName "kube-api-access-68m56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:40.844637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.844567    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49286968-069b-4864-a7bc-aedbb7d8a258-kube-api-access-zn8n8" pod "49286968-069b-4864-a7bc-aedbb7d8a258" (UID: "49286968-069b-4864-a7bc-aedbb7d8a258"). InnerVolumeSpecName "kube-api-access-zn8n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:40.944803 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.944707    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn8n8\" (UniqueName: \"kubernetes.io/projected/49286968-069b-4864-a7bc-aedbb7d8a258-kube-api-access-zn8n8\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:40.944803 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:40.944745    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68m56\" (UniqueName: \"kubernetes.io/projected/3f623bbd-bc79-4756-b7fb-f4d656ebddff-kube-api-access-68m56\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:41.043963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.043921    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:41.675719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.675684    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"
Nov 03 20:05:41.675719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.675699    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj" event={"ID":"49286968-069b-4864-a7bc-aedbb7d8a258","Type":"ContainerDied","Data":"ae7174de33746de9b34fc36491a6525510105a649d5b8ecb8ae311296676c909"}
Nov 03 20:05:41.675971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.675767    5659 scope.go:122] "RemoveContainer" containerID="8e154b611c9aa565536d5612f4efdff490b5ac936639a72999f7726f1fa87789"
Nov 03 20:05:41.679373 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.679345    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf" event={"ID":"3f623bbd-bc79-4756-b7fb-f4d656ebddff","Type":"ContainerDied","Data":"c21ca159863451c15c81be214fe176b5af5b7376ac0b58cfcdbea0e524bc404f"}
Nov 03 20:05:41.679530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.679440    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"
Nov 03 20:05:41.687860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.687841    5659 scope.go:122] "RemoveContainer" containerID="ab79d9804869c9cd74535af043c1601ab08642e5972f2b7b5ef50ba21bf66434"
Nov 03 20:05:41.707949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.707927    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"]
Nov 03 20:05:41.712406 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.712382    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-78d47a-66f8c9595-xvsjj"]
Nov 03 20:05:41.722882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.722861    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"]
Nov 03 20:05:41.727472 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.727451    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-78d47a-568c4fc5dd-qjxxf"]
Nov 03 20:05:41.809702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.805350    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:41.968548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.968515    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f623bbd-bc79-4756-b7fb-f4d656ebddff" path="/var/lib/kubelet/pods/3f623bbd-bc79-4756-b7fb-f4d656ebddff/volumes"
Nov 03 20:05:41.968941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.968922    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49286968-069b-4864-a7bc-aedbb7d8a258" path="/var/lib/kubelet/pods/49286968-069b-4864-a7bc-aedbb7d8a258/volumes"
Nov 03 20:05:41.969331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:41.969319    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a80a4e5e-576a-4933-9e86-bb68d58cd9db" path="/var/lib/kubelet/pods/a80a4e5e-576a-4933-9e86-bb68d58cd9db/volumes"
Nov 03 20:05:42.518308 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:42.518272    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:43.211279 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:43.211227    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:43.344041 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:43.344008    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:43.738903 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:43.738851    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:44.140367 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:44.140282    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:44.299504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:44.299103    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:44.822361 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:44.822329    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:45.056924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.055844    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:45.219729 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:05:45.219681    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:05:45.288840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.287443    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"]
Nov 03 20:05:45.288840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.288270    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3f623bbd-bc79-4756-b7fb-f4d656ebddff" containerName="payload-processing"
Nov 03 20:05:45.288840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.288292    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f623bbd-bc79-4756-b7fb-f4d656ebddff" containerName="payload-processing"
Nov 03 20:05:45.288840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.288329    5659 cpu_manager.go:414] "containerMap: removing container" podUID="49286968-069b-4864-a7bc-aedbb7d8a258" containerName="payload-pre-processing"
Nov 03 20:05:45.288840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.288338    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49286968-069b-4864-a7bc-aedbb7d8a258" containerName="payload-pre-processing"
Nov 03 20:05:45.288840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.288364    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a80a4e5e-576a-4933-9e86-bb68d58cd9db" containerName="maas-api"
Nov 03 20:05:45.288840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.288372    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80a4e5e-576a-4933-9e86-bb68d58cd9db" containerName="maas-api"
Nov 03 20:05:45.288840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.288568    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f623bbd-bc79-4756-b7fb-f4d656ebddff" containerName="payload-processing"
Nov 03 20:05:45.288840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.288587    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="49286968-069b-4864-a7bc-aedbb7d8a258" containerName="payload-pre-processing"
Nov 03 20:05:45.288840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.288601    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80a4e5e-576a-4933-9e86-bb68d58cd9db" containerName="maas-api"
Nov 03 20:05:45.294653 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.294631    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.298065 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.298042    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"]
Nov 03 20:05:45.301087 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.301065    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-a44263-openshift-default-dockercfg-rh7n7\""
Nov 03 20:05:45.385506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.385448    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:45.423748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.423712    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-envoy\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.423902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.423802    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-token\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.423902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.423847    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-data\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.424053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.423912    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-credential-socket\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.424053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.423975    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-socket\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.424053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.424032    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-certs\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.424164 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.424139    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc6a816-f845-472c-b39f-cb61910c8d92-istiod-ca-cert\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.424224 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.424211    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-podinfo\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.424365 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.424343    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pgdd\" (UniqueName: \"kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-kube-api-access-2pgdd\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.548674 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.548561    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-podinfo\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.548674 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.548647    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pgdd\" (UniqueName: \"kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-kube-api-access-2pgdd\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.548913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.548686    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-envoy\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.548913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.548732    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-token\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.548913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.548784    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-data\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.548913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.548819    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-credential-socket\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.548913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.548887    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-socket\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.549192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.548940    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-certs\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.549192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.549073    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc6a816-f845-472c-b39f-cb61910c8d92-istiod-ca-cert\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.553506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.549902    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc6a816-f845-472c-b39f-cb61910c8d92-istiod-ca-cert\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.553506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.550842    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-data\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.553506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.550895    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-credential-socket\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.553506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.551454    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-certs\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.553506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.551772    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-socket\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.556904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.556870    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-podinfo\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.577505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.571357    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-envoy\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.618502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.618236    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pgdd\" (UniqueName: \"kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-kube-api-access-2pgdd\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.618707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.618671    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-token\") pod \"e2e-noten-a44263-openshift-default-668c89cc89-vx5cj\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") " pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.622514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.619634    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:45.646850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:45.646775    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:46.051350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:46.048940    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"]
Nov 03 20:05:46.062728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:46.061553    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:46.104546 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:05:46.099122    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc6a816_f845_472c_b39f_cb61910c8d92.slice/crio-4e5872950af1a49d43166ade743ffa7d3e1ab34b9f38b1a6580e097cc6935482 WatchSource:0}: Error finding container 4e5872950af1a49d43166ade743ffa7d3e1ab34b9f38b1a6580e097cc6935482: Status 404 returned error can't find the container with id 4e5872950af1a49d43166ade743ffa7d3e1ab34b9f38b1a6580e097cc6935482
Nov 03 20:05:46.110915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:46.110865    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:46.111075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:46.110973    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:46.111075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:46.111019    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:46.729517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:46.728304    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:46.786245 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:46.782977    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj" event={"ID":"3fc6a816-f845-472c-b39f-cb61910c8d92","Type":"ContainerStarted","Data":"ee2d7e847327aa29d8dc10221395dd473c090c28f993dd338bfcc2a00a79ee5c"}
Nov 03 20:05:46.786245 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:46.783119    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj" event={"ID":"3fc6a816-f845-472c-b39f-cb61910c8d92","Type":"ContainerStarted","Data":"4e5872950af1a49d43166ade743ffa7d3e1ab34b9f38b1a6580e097cc6935482"}
Nov 03 20:05:46.813305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:46.813254    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj" podStartSLOduration=1.8132384099999999 podStartE2EDuration="1.81323841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:46.810237177 +0000 UTC m=+1698.965139620" watchObservedRunningTime="2026-11-03 20:05:46.81323841 +0000 UTC m=+1698.968140829"
Nov 03 20:05:47.193324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:47.193243    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:47.333813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:47.333773    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:47.505154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:47.505114    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:47.620450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:47.620416    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:47.627357 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:47.627327    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:47.647040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:47.647003    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:47.790716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:47.790625    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:47.791747 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:47.791725    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:05:48.470952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:48.470916    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:48.606793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:48.606758    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:49.346278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:49.346233    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:50.049503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:50.049450    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:50.869001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:50.868856    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:51.683517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:51.677688    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:52.386555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:52.386207    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:53.355104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:53.355072    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:54.250375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:54.250338    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:55.151033 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:55.150998    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:55.974045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:55.973995    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:56.727784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:56.726108    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:57.523194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:57.523113    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:58.422283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:58.422250    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:59.152083 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:59.152043    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:59.842747 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:05:59.842708    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:00.158249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:00.158125    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:00.173844 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:00.173782    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"]
Nov 03 20:06:00.174746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:00.174647    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj" podUID="3fc6a816-f845-472c-b39f-cb61910c8d92" containerName="istio-proxy" containerID="cri-o://ee2d7e847327aa29d8dc10221395dd473c090c28f993dd338bfcc2a00a79ee5c" gracePeriod=30
Nov 03 20:06:00.306508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:00.301390    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:00.447562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:00.446450    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:00.644171 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:00.644137    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:00.875093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:00.875057    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:01.548523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:01.547455    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:02.250720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:02.250685    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:02.792705 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:02.792659    5659 patch_prober.go:29] interesting pod/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:06:02.793105 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:02.792732    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj" podUID="3fc6a816-f845-472c-b39f-cb61910c8d92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:06:02.945731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:02.945702    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:03.640477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:03.640441    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:04.340627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:04.340564    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:05.176644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:05.176607    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:05.630184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:05.630150    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:05.944253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:05.944212    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:06:05.945222 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:05.945188    5659 generic.go:356] "Generic (PLEG): container finished" podID="3fc6a816-f845-472c-b39f-cb61910c8d92" containerID="ee2d7e847327aa29d8dc10221395dd473c090c28f993dd338bfcc2a00a79ee5c" exitCode=0
Nov 03 20:06:05.945363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:05.945311    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj" event={"ID":"3fc6a816-f845-472c-b39f-cb61910c8d92","Type":"ContainerDied","Data":"ee2d7e847327aa29d8dc10221395dd473c090c28f993dd338bfcc2a00a79ee5c"}
Nov 03 20:06:05.945410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:05.945391    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj" event={"ID":"3fc6a816-f845-472c-b39f-cb61910c8d92","Type":"ContainerDied","Data":"4e5872950af1a49d43166ade743ffa7d3e1ab34b9f38b1a6580e097cc6935482"}
Nov 03 20:06:05.945457 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:05.945420    5659 scope.go:122] "RemoveContainer" containerID="ee2d7e847327aa29d8dc10221395dd473c090c28f993dd338bfcc2a00a79ee5c"
Nov 03 20:06:05.958828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:05.958800    5659 scope.go:122] "RemoveContainer" containerID="ee2d7e847327aa29d8dc10221395dd473c090c28f993dd338bfcc2a00a79ee5c"
Nov 03 20:06:05.959118 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:06:05.959100    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2d7e847327aa29d8dc10221395dd473c090c28f993dd338bfcc2a00a79ee5c\": container with ID starting with ee2d7e847327aa29d8dc10221395dd473c090c28f993dd338bfcc2a00a79ee5c not found: ID does not exist" containerID="ee2d7e847327aa29d8dc10221395dd473c090c28f993dd338bfcc2a00a79ee5c"
Nov 03 20:06:05.959200 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:05.959128    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2d7e847327aa29d8dc10221395dd473c090c28f993dd338bfcc2a00a79ee5c"} err="failed to get container status \"ee2d7e847327aa29d8dc10221395dd473c090c28f993dd338bfcc2a00a79ee5c\": rpc error: code = NotFound desc = could not find container \"ee2d7e847327aa29d8dc10221395dd473c090c28f993dd338bfcc2a00a79ee5c\": container with ID starting with ee2d7e847327aa29d8dc10221395dd473c090c28f993dd338bfcc2a00a79ee5c not found: ID does not exist"
Nov 03 20:06:06.021799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.021757    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-envoy\") pod \"3fc6a816-f845-472c-b39f-cb61910c8d92\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") "
Nov 03 20:06:06.021965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.021842    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc6a816-f845-472c-b39f-cb61910c8d92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc6a816-f845-472c-b39f-cb61910c8d92-istiod-ca-cert\") pod \"3fc6a816-f845-472c-b39f-cb61910c8d92\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") "
Nov 03 20:06:06.021965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.021898    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-socket\") pod \"3fc6a816-f845-472c-b39f-cb61910c8d92\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") "
Nov 03 20:06:06.021965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.021930    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-podinfo\") pod \"3fc6a816-f845-472c-b39f-cb61910c8d92\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") "
Nov 03 20:06:06.021965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.021963    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-data\") pod \"3fc6a816-f845-472c-b39f-cb61910c8d92\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") "
Nov 03 20:06:06.022136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.022045    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-kube-api-access-2pgdd\" (UniqueName: \"kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-kube-api-access-2pgdd\") pod \"3fc6a816-f845-472c-b39f-cb61910c8d92\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") "
Nov 03 20:06:06.022136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.022121    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-credential-socket\") pod \"3fc6a816-f845-472c-b39f-cb61910c8d92\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") "
Nov 03 20:06:06.022199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.022162    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-token\" (UniqueName: \"kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-token\") pod \"3fc6a816-f845-472c-b39f-cb61910c8d92\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") "
Nov 03 20:06:06.022363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.022335    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-certs\") pod \"3fc6a816-f845-472c-b39f-cb61910c8d92\" (UID: \"3fc6a816-f845-472c-b39f-cb61910c8d92\") "
Nov 03 20:06:06.022849 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.022820    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-data" pod "3fc6a816-f845-472c-b39f-cb61910c8d92" (UID: "3fc6a816-f845-472c-b39f-cb61910c8d92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:06.023162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.023139    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc6a816-f845-472c-b39f-cb61910c8d92-istiod-ca-cert" pod "3fc6a816-f845-472c-b39f-cb61910c8d92" (UID: "3fc6a816-f845-472c-b39f-cb61910c8d92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:06.023238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.023142    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.023387 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.023366    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-socket" pod "3fc6a816-f845-472c-b39f-cb61910c8d92" (UID: "3fc6a816-f845-472c-b39f-cb61910c8d92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:06.025152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.024591    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-envoy" pod "3fc6a816-f845-472c-b39f-cb61910c8d92" (UID: "3fc6a816-f845-472c-b39f-cb61910c8d92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:06.025152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.024999    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-certs" pod "3fc6a816-f845-472c-b39f-cb61910c8d92" (UID: "3fc6a816-f845-472c-b39f-cb61910c8d92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:06.025152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.025052    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-credential-socket" pod "3fc6a816-f845-472c-b39f-cb61910c8d92" (UID: "3fc6a816-f845-472c-b39f-cb61910c8d92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:06.029210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.027790    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-kube-api-access-2pgdd" pod "3fc6a816-f845-472c-b39f-cb61910c8d92" (UID: "3fc6a816-f845-472c-b39f-cb61910c8d92"). InnerVolumeSpecName "kube-api-access-2pgdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:06.029210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.027944    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-podinfo" pod "3fc6a816-f845-472c-b39f-cb61910c8d92" (UID: "3fc6a816-f845-472c-b39f-cb61910c8d92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:06:06.029210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.028923    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-token" pod "3fc6a816-f845-472c-b39f-cb61910c8d92" (UID: "3fc6a816-f845-472c-b39f-cb61910c8d92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:06.124369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.124334    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.124369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.124364    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc6a816-f845-472c-b39f-cb61910c8d92-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.124369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.124374    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.124631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.124383    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.124631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.124392    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pgdd\" (UniqueName: \"kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-kube-api-access-2pgdd\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.124631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.124403    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.124631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.124411    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc6a816-f845-472c-b39f-cb61910c8d92-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.124631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.124419    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc6a816-f845-472c-b39f-cb61910c8d92-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.181291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.181173    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:06.842029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.841988    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:06.953186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.953157    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"
Nov 03 20:06:06.983528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.983462    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"]
Nov 03 20:06:06.986863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:06.986837    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-a44263-openshift-default-668c89cc89-vx5cj"]
Nov 03 20:06:07.706540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:07.701992    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:07.977915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:07.977831    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc6a816-f845-472c-b39f-cb61910c8d92" path="/var/lib/kubelet/pods/3fc6a816-f845-472c-b39f-cb61910c8d92/volumes"
Nov 03 20:06:08.388768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:08.388675    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:09.158097 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:09.158015    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:09.887543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:09.883843    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:10.771512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:10.759718    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:11.554076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:11.554043    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:12.338929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:12.338898    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:13.034604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:13.034567    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:13.747722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:13.747684    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:14.430385 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:14.430355    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:15.146401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:15.146321    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:15.230594 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:06:15.230463    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:06:15.868514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:15.866920    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:16.635285 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:16.635251    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:17.290990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:17.290957    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:17.951558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:17.951510    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:18.753113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:18.753064    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:19.444524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:19.444466    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:20.126585 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:20.126504    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:20.862212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:20.862180    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:21.647364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:21.647328    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:22.255798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:22.255762    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:22.960739 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:22.960705    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:23.651403 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:23.651342    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:24.437568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:24.437534    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:25.241968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:25.241935    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:25.966172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:25.966137    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:26.752424 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:26.752384    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:27.968789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:27.968760    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:28.201638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:28.201597    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:28.895692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:28.895644    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:29.833583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:29.833538    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:30.561549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:30.561512    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:31.243108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:31.243072    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:32.053826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:32.053787    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:32.757245 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:32.757211    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:33.494153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:33.494119    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:34.441704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:34.441660    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:35.154622 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:35.154580    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:35.832706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:35.832672    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:36.657590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:36.657557    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:36.935608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:36.935510    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:37.537183 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:37.537140    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:38.220363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:38.220333    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:38.973477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:38.973432    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:39.679630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:39.679596    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:40.359518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:40.350988    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:41.037440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:41.037397    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:41.846061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:41.846026    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:42.778299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:42.778253    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:43.410059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:43.410019    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:43.732235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:43.732197    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:44.429402 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:44.429366    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:45.129587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:45.129551    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:45.222532 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:06:45.222497    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:06:45.899214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:45.899183    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:46.642850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:46.642805    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:47.285620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.285582    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:47.427436 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.427401    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:47.694767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.694683    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"]
Nov 03 20:06:47.695691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.695662    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3fc6a816-f845-472c-b39f-cb61910c8d92" containerName="istio-proxy"
Nov 03 20:06:47.695691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.695690    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc6a816-f845-472c-b39f-cb61910c8d92" containerName="istio-proxy"
Nov 03 20:06:47.695892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.695872    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc6a816-f845-472c-b39f-cb61910c8d92" containerName="istio-proxy"
Nov 03 20:06:47.701631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.701605    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.708703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.708672    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-836265-openshift-default-dockercfg-t9cxz\""
Nov 03 20:06:47.714516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.714456    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"]
Nov 03 20:06:47.763570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.763524    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-certs\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.763772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.763634    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-credential-socket\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.763772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.763735    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqvq\" (UniqueName: \"kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-kube-api-access-2gqvq\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.763905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.763892    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-podinfo\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.764026 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.764000    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8596f4f0-3ef7-446c-8374-57bcf455301c-istiod-ca-cert\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.764115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.764090    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-data\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.764176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.764138    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-envoy\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.764237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.764169    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-socket\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.764299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.764238    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-token\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.775853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.775821    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:47.878503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.868643    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-podinfo\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.878503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.868756    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8596f4f0-3ef7-446c-8374-57bcf455301c-istiod-ca-cert\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.878503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.868827    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-data\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.878503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.868861    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-envoy\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.878503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.868891    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-socket\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.878503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.868942    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-token\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.878503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.868996    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-certs\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.878503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.869066    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-credential-socket\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.878503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.869139    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqvq\" (UniqueName: \"kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-kube-api-access-2gqvq\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.892394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.891071    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-socket\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.892394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.891817    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8596f4f0-3ef7-446c-8374-57bcf455301c-istiod-ca-cert\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.892394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.892071    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-data\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.893046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.893023    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-certs\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.893767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.893743    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-credential-socket\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.920535 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.917867    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-envoy\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.925166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.925133    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-podinfo\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.939620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.939588    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqvq\" (UniqueName: \"kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-kube-api-access-2gqvq\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:47.954153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:47.954042    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-token\") pod \"e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") " pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:48.037052 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:48.037019    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:48.068643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:48.068561    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:48.273755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:48.273664    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:48.407345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:48.407314    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"]
Nov 03 20:06:48.409903 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:06:48.409874    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8596f4f0_3ef7_446c_8374_57bcf455301c.slice/crio-d586b54f0ffe303b888360c1e0db5570da5d234df4bdce5768cd3de3e92a24e4 WatchSource:0}: Error finding container d586b54f0ffe303b888360c1e0db5570da5d234df4bdce5768cd3de3e92a24e4: Status 404 returned error can't find the container with id d586b54f0ffe303b888360c1e0db5570da5d234df4bdce5768cd3de3e92a24e4
Nov 03 20:06:48.412665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:48.412628    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:06:48.412762 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:48.412700    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:06:48.412762 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:48.412729    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:06:48.963692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:48.963656    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:49.253560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:49.253459    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w" event={"ID":"8596f4f0-3ef7-446c-8374-57bcf455301c","Type":"ContainerStarted","Data":"4a5260a0b8cc18c20c0f6d1dd154cbc65cfd5b01e5c7c4b8497c326a357089f5"}
Nov 03 20:06:49.253560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:49.253535    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w" event={"ID":"8596f4f0-3ef7-446c-8374-57bcf455301c","Type":"ContainerStarted","Data":"d586b54f0ffe303b888360c1e0db5570da5d234df4bdce5768cd3de3e92a24e4"}
Nov 03 20:06:49.285380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:49.285320    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w" podStartSLOduration=2.285306302 podStartE2EDuration="2.285306302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:49.279957995 +0000 UTC m=+1761.434860426" watchObservedRunningTime="2026-11-03 20:06:49.285306302 +0000 UTC m=+1761.440208721"
Nov 03 20:06:49.750839 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:49.750777    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:50.038206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:50.038106    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:50.046927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:50.046896    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:50.263004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:50.262975    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:50.264194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:50.264167    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:06:50.487636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:50.482932    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:51.195949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:51.195919    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:51.608461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:51.608407    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:51.753711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:51.753675    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:52.040897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:52.040861    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:52.852596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:52.852566    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:52.947148 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:52.947114    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"]
Nov 03 20:06:52.969938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:52.969908    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"]
Nov 03 20:06:52.970178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:52.970008    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"
Nov 03 20:06:53.014864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:53.014815    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:53.030028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:53.018954    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-836265-dockercfg-qmqcx\""
Nov 03 20:06:53.049350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:53.031214    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-836265\""
Nov 03 20:06:53.051978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:53.051132    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4dzm\" (UniqueName: \"kubernetes.io/projected/3079f0fd-1225-42ab-8151-4062129bcca0-kube-api-access-l4dzm\") pod \"payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq\" (UID: \"3079f0fd-1225-42ab-8151-4062129bcca0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"
Nov 03 20:06:53.051978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:53.051371    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3079f0fd-1225-42ab-8151-4062129bcca0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq\" (UID: \"3079f0fd-1225-42ab-8151-4062129bcca0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"
Nov 03 20:06:53.166658 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:53.166563    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4dzm\" (UniqueName: \"kubernetes.io/projected/3079f0fd-1225-42ab-8151-4062129bcca0-kube-api-access-l4dzm\") pod \"payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq\" (UID: \"3079f0fd-1225-42ab-8151-4062129bcca0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"
Nov 03 20:06:53.166834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:53.166757    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3079f0fd-1225-42ab-8151-4062129bcca0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq\" (UID: \"3079f0fd-1225-42ab-8151-4062129bcca0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"
Nov 03 20:06:53.167650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:53.167625    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3079f0fd-1225-42ab-8151-4062129bcca0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq\" (UID: \"3079f0fd-1225-42ab-8151-4062129bcca0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"
Nov 03 20:06:53.198373 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:53.196217    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4dzm\" (UniqueName: \"kubernetes.io/projected/3079f0fd-1225-42ab-8151-4062129bcca0-kube-api-access-l4dzm\") pod \"payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq\" (UID: \"3079f0fd-1225-42ab-8151-4062129bcca0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"
Nov 03 20:06:53.291714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:53.291683    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"
Nov 03 20:06:53.587337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:53.587297    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"]
Nov 03 20:06:53.587680 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:06:53.587657    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3079f0fd_1225_42ab_8151_4062129bcca0.slice/crio-d378adaa513bcb6fc1faaf8394801f69770d5fecc9b88844878d1726d5e867c2 WatchSource:0}: Error finding container d378adaa513bcb6fc1faaf8394801f69770d5fecc9b88844878d1726d5e867c2: Status 404 returned error can't find the container with id d378adaa513bcb6fc1faaf8394801f69770d5fecc9b88844878d1726d5e867c2
Nov 03 20:06:53.645326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:53.645298    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:54.296813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:54.296782    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq" event={"ID":"3079f0fd-1225-42ab-8151-4062129bcca0","Type":"ContainerStarted","Data":"4957526e2ac76067ecfc42871eb8fa8794ac687c8ba7601061721a231ed6e926"}
Nov 03 20:06:54.297211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:54.296831    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq" event={"ID":"3079f0fd-1225-42ab-8151-4062129bcca0","Type":"ContainerStarted","Data":"d378adaa513bcb6fc1faaf8394801f69770d5fecc9b88844878d1726d5e867c2"}
Nov 03 20:06:54.297211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:54.296959    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"
Nov 03 20:06:54.317378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:54.317323    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq" podStartSLOduration=2.317309671 podStartE2EDuration="2.317309671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:54.315132924 +0000 UTC m=+1766.470035344" watchObservedRunningTime="2026-11-03 20:06:54.317309671 +0000 UTC m=+1766.472212119"
Nov 03 20:06:54.343437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:54.343409    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:55.309824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:55.309788    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:55.916512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:55.916217    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:56.685526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:56.684873    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:57.472119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:57.472083    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:58.207704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:58.207600    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:58.870681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:58.870647    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:59.551464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:06:59.551434    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:00.262957 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:00.262922    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:00.936643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:00.936613    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:01.073553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:01.073463    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:01.230408 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:01.230374    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:01.931285 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:01.931253    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:02.909707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:02.909661    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:03.764705 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:03.764666    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:04.447677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:04.447636    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:05.220441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:05.220409    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:05.310634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:05.310602    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"
Nov 03 20:07:05.539792 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:05.539702    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:05.991367 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:05.991331    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:06.649187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:06.649146    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:07.388923 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:07.388644    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:08.049163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:08.049119    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:08.748942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:08.748893    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:09.441612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:09.441580    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:10.169045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:10.169013    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:11.228720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:11.228683    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:11.919137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:11.919097    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:12.658386 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:12.658341    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:13.603512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:13.601090    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:14.335843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:14.335807    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:15.005312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:15.005277    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:15.219790 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:07:15.219758    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:07:15.768978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:15.768948    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:16.537027 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:16.536985    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:16.758602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:16.758562    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:17.186166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:17.186117    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:17.334423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:17.334380    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:17.758581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:17.758536    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:17.946664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:17.946579    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:18.371147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:18.371113    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:19.182507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:19.180472    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:19.966197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:19.965458    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:20.387467 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:20.387388    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:20.850533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:20.848978    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:21.281557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:21.281511    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:21.450009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:21.449937    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:21.756869 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:21.756833    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:22.451754 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.451649    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:22.511733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.511699    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"]
Nov 03 20:07:22.515751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.515727    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"
Nov 03 20:07:22.522659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.522633    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-660d95\""
Nov 03 20:07:22.525685 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.525663    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-660d95-dockercfg-f4n88\""
Nov 03 20:07:22.527300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.527277    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"]
Nov 03 20:07:22.570444 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.570406    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"]
Nov 03 20:07:22.575356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.575329    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"
Nov 03 20:07:22.581965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.581936    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:22.593904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.593880    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"]
Nov 03 20:07:22.723505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.720557    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376c36bb-8e35-46d3-9ced-b2a66f7370f7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc\" (UID: \"376c36bb-8e35-46d3-9ced-b2a66f7370f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"
Nov 03 20:07:22.723505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.720735    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf0a278-c34b-484d-9b5e-11243d3c2c63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx\" (UID: \"fbf0a278-c34b-484d-9b5e-11243d3c2c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"
Nov 03 20:07:22.723505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.720805    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwghp\" (UniqueName: \"kubernetes.io/projected/fbf0a278-c34b-484d-9b5e-11243d3c2c63-kube-api-access-qwghp\") pod \"payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx\" (UID: \"fbf0a278-c34b-484d-9b5e-11243d3c2c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"
Nov 03 20:07:22.723505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.720856    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl9zs\" (UniqueName: \"kubernetes.io/projected/376c36bb-8e35-46d3-9ced-b2a66f7370f7-kube-api-access-fl9zs\") pod \"payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc\" (UID: \"376c36bb-8e35-46d3-9ced-b2a66f7370f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"
Nov 03 20:07:22.832503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.830465    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf0a278-c34b-484d-9b5e-11243d3c2c63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx\" (UID: \"fbf0a278-c34b-484d-9b5e-11243d3c2c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"
Nov 03 20:07:22.832503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.830588    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwghp\" (UniqueName: \"kubernetes.io/projected/fbf0a278-c34b-484d-9b5e-11243d3c2c63-kube-api-access-qwghp\") pod \"payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx\" (UID: \"fbf0a278-c34b-484d-9b5e-11243d3c2c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"
Nov 03 20:07:22.835400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.834889    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl9zs\" (UniqueName: \"kubernetes.io/projected/376c36bb-8e35-46d3-9ced-b2a66f7370f7-kube-api-access-fl9zs\") pod \"payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc\" (UID: \"376c36bb-8e35-46d3-9ced-b2a66f7370f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"
Nov 03 20:07:22.835400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.835086    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376c36bb-8e35-46d3-9ced-b2a66f7370f7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc\" (UID: \"376c36bb-8e35-46d3-9ced-b2a66f7370f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"
Nov 03 20:07:22.848551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.836467    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376c36bb-8e35-46d3-9ced-b2a66f7370f7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc\" (UID: \"376c36bb-8e35-46d3-9ced-b2a66f7370f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"
Nov 03 20:07:22.848551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.837256    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf0a278-c34b-484d-9b5e-11243d3c2c63-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx\" (UID: \"fbf0a278-c34b-484d-9b5e-11243d3c2c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"
Nov 03 20:07:22.874504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.871510    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl9zs\" (UniqueName: \"kubernetes.io/projected/376c36bb-8e35-46d3-9ced-b2a66f7370f7-kube-api-access-fl9zs\") pod \"payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc\" (UID: \"376c36bb-8e35-46d3-9ced-b2a66f7370f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"
Nov 03 20:07:22.878510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.875253    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwghp\" (UniqueName: \"kubernetes.io/projected/fbf0a278-c34b-484d-9b5e-11243d3c2c63-kube-api-access-qwghp\") pod \"payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx\" (UID: \"fbf0a278-c34b-484d-9b5e-11243d3c2c63\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"
Nov 03 20:07:22.896317 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:22.894945    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"
Nov 03 20:07:23.133769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:23.133678    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"
Nov 03 20:07:23.193347 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:23.193316    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"]
Nov 03 20:07:23.300046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:23.299974    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:23.428103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:23.428074    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"]
Nov 03 20:07:23.428640 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:07:23.428616    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf0a278_c34b_484d_9b5e_11243d3c2c63.slice/crio-193bcd2f715278e40bcf4db7606ce6dd6c52b89900f207af3b1aedfa02ad0406 WatchSource:0}: Error finding container 193bcd2f715278e40bcf4db7606ce6dd6c52b89900f207af3b1aedfa02ad0406: Status 404 returned error can't find the container with id 193bcd2f715278e40bcf4db7606ce6dd6c52b89900f207af3b1aedfa02ad0406
Nov 03 20:07:23.551524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:23.543191    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx" event={"ID":"fbf0a278-c34b-484d-9b5e-11243d3c2c63","Type":"ContainerStarted","Data":"c239731ac9c2e45cd9c00b0f6862a4129ed9c55588a868ccf1d5eab48f97e263"}
Nov 03 20:07:23.551524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:23.543261    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx" event={"ID":"fbf0a278-c34b-484d-9b5e-11243d3c2c63","Type":"ContainerStarted","Data":"193bcd2f715278e40bcf4db7606ce6dd6c52b89900f207af3b1aedfa02ad0406"}
Nov 03 20:07:23.551524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:23.544363    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"
Nov 03 20:07:23.551524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:23.550281    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc" event={"ID":"376c36bb-8e35-46d3-9ced-b2a66f7370f7","Type":"ContainerStarted","Data":"35887100fca9ea6d3ba2319a22b918fab53f2b0f873cb23f2f12cbc4027e3d4f"}
Nov 03 20:07:23.551524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:23.550329    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc" event={"ID":"376c36bb-8e35-46d3-9ced-b2a66f7370f7","Type":"ContainerStarted","Data":"136ce1abe3627ce65b8feb438eb4f6f5351ea4f8af69fad3a6491789af294420"}
Nov 03 20:07:23.551524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:23.550569    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"
Nov 03 20:07:23.567509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:23.567433    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx" podStartSLOduration=1.5674171559999999 podStartE2EDuration="1.567417156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:23.563043307 +0000 UTC m=+1795.717945728" watchObservedRunningTime="2026-11-03 20:07:23.567417156 +0000 UTC m=+1795.722319576"
Nov 03 20:07:23.583151 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:23.583094    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc" podStartSLOduration=1.583066349 podStartE2EDuration="1.583066349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:23.581413195 +0000 UTC m=+1795.736315617" watchObservedRunningTime="2026-11-03 20:07:23.583066349 +0000 UTC m=+1795.737968768"
Nov 03 20:07:24.160220 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:24.160179    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:24.848007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:24.847971    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:25.525163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:25.525127    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:26.164083 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:26.164036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:26.850777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:26.850710    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:27.540680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:27.540648    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:28.265016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:28.260919    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:29.193503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:29.181166    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:29.998405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:29.998370    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:30.773620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:30.773587    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:31.563373 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:31.563334    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:32.257935 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:32.257894    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:32.398989 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:32.398953    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:33.198632 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:33.198597    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:34.113933 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:34.113901    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:34.565873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:34.565831    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"
Nov 03 20:07:34.947094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:34.947016    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:35.602424 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:35.602393    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"
Nov 03 20:07:35.665514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:35.658175    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:36.398808 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:36.398771    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:36.992874 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:36.992824    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:37.125078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:37.125045    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:37.748835 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:37.748799    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:38.441314 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:38.441279    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:39.213390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:39.213255    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:39.344368 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:39.344331    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:39.940834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:39.940803    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:40.671245 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:40.671127    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:40.770881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:40.770849    5659 generic.go:356] "Generic (PLEG): container finished" podID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerID="4a5260a0b8cc18c20c0f6d1dd154cbc65cfd5b01e5c7c4b8497c326a357089f5" exitCode=0
Nov 03 20:07:40.771061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:40.770910    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w" event={"ID":"8596f4f0-3ef7-446c-8374-57bcf455301c","Type":"ContainerDied","Data":"4a5260a0b8cc18c20c0f6d1dd154cbc65cfd5b01e5c7c4b8497c326a357089f5"}
Nov 03 20:07:40.771432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:40.771396    5659 scope.go:122] "RemoveContainer" containerID="4a5260a0b8cc18c20c0f6d1dd154cbc65cfd5b01e5c7c4b8497c326a357089f5"
Nov 03 20:07:40.773303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:40.773272    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:07:40.773391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:40.773347    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:07:40.773391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:40.773380    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:07:41.459160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:41.459126    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:41.780508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:41.780398    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w" event={"ID":"8596f4f0-3ef7-446c-8374-57bcf455301c","Type":"ContainerStarted","Data":"53685b48da32a7f271eb55f633edeb36f59e20e79499279cbb0766d9cc11ebf2"}
Nov 03 20:07:42.038386 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:42.037879    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:07:42.038386 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:42.038011    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 20:07:42.038386 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:42.038057    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w" podUID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 20:07:42.144010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:42.143976    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:42.790226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:42.790194    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:43.037881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:43.037841    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 20:07:43.038168 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:43.038137    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w" podUID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 20:07:43.562505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:43.561962    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:44.040912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:44.040874    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 20:07:44.042343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:44.042310    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w" podUID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 20:07:44.237274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:44.237237    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:45.042670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:45.042635    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 20:07:45.043281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:45.043246    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w" podUID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 20:07:45.153193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:45.153156    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:45.266950 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:07:45.266822    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:07:45.866808 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:45.866777    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:46.041637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:46.041604    5659 patch_prober.go:29] interesting pod/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 20:07:46.041925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:46.041894    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w" podUID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 20:07:46.560952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:46.560916    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:47.102346 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:47.102312    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:07:47.103886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:47.103855    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:07:47.106618 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:47.106591    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:07:47.331733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:47.331700    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:48.142413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:48.142380    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:48.951545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:48.947970    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:49.624897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:49.624865    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:50.281597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:50.281566    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:51.039441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:51.039409    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:51.426811 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:51.426726    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:51.759647 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:51.759616    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:52.557865 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:52.557833    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:53.269508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:53.265984    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:54.122355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:54.122322    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:54.829617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:54.829580    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:55.723593 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:55.723560    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:56.489587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:56.473448    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:57.133888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:57.133848    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:57.952250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:57.952217    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:58.745941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:58.745903    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:59.412126 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:07:59.412098    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:00.064009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:00.063972    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:00.784019 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:00.783926    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:01.784722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:01.784681    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:02.865345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:02.865309    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:03.628939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:03.628897    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:04.364057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:04.363984    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:05.112001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:05.111973    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:06.025867 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:06.025836    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:06.750586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:06.750539    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:07.435888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:07.435848    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:08.085583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:08.085543    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:08.821719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:08.821676    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:09.656628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:09.656595    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:10.470960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:10.470856    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:11.235218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:11.235185    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:11.947521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:11.947468    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:12.087137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:12.087106    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:12.424513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:12.424466    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:12.573194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:12.573153    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:13.038345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:13.038297    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:13.870026 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:13.869986    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:14.737946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:14.737908    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:15.223176 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:08:15.223138    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:08:15.738400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:15.738360    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:16.329860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:16.329832    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:16.464002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:16.463964    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:16.850498 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:16.850453    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:17.519843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.519813    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:17.569121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.569086    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"]
Nov 03 20:08:17.590047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.590012    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"]
Nov 03 20:08:17.590233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.590145    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"
Nov 03 20:08:17.597403 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.597153    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-eb69cb-dockercfg-mzm8r\""
Nov 03 20:08:17.622331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.622279    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-976wz\" (UniqueName: \"kubernetes.io/projected/cce0df39-f9a8-4347-aa52-423d78661bfe-kube-api-access-976wz\") pod \"payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd\" (UID: \"cce0df39-f9a8-4347-aa52-423d78661bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"
Nov 03 20:08:17.622833 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.622789    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce0df39-f9a8-4347-aa52-423d78661bfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd\" (UID: \"cce0df39-f9a8-4347-aa52-423d78661bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"
Nov 03 20:08:17.649826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.649793    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-eb69cb\""
Nov 03 20:08:17.699988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.699955    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:17.725745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.725703    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-976wz\" (UniqueName: \"kubernetes.io/projected/cce0df39-f9a8-4347-aa52-423d78661bfe-kube-api-access-976wz\") pod \"payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd\" (UID: \"cce0df39-f9a8-4347-aa52-423d78661bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"
Nov 03 20:08:17.729322 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.726420    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce0df39-f9a8-4347-aa52-423d78661bfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd\" (UID: \"cce0df39-f9a8-4347-aa52-423d78661bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"
Nov 03 20:08:17.733750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.730375    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce0df39-f9a8-4347-aa52-423d78661bfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd\" (UID: \"cce0df39-f9a8-4347-aa52-423d78661bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"
Nov 03 20:08:17.760244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.760197    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-976wz\" (UniqueName: \"kubernetes.io/projected/cce0df39-f9a8-4347-aa52-423d78661bfe-kube-api-access-976wz\") pod \"payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd\" (UID: \"cce0df39-f9a8-4347-aa52-423d78661bfe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"
Nov 03 20:08:17.804562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.804436    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"]
Nov 03 20:08:17.854636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.854603    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"]
Nov 03 20:08:17.855285 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.855211    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:17.900542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.899064    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-660d95-openshift-default-dockercfg-df2mh\""
Nov 03 20:08:17.912977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.912928    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"
Nov 03 20:08:17.937521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.937462    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:17.937633 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.937569    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:17.937724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.937685    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:17.937847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.937829    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:17.937951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.937867    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:17.937951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.937941    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r5ff\" (UniqueName: \"kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-kube-api-access-7r5ff\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:17.938178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.938056    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:17.938178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.938126    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c85f57d-183f-408c-a450-6d6ad119a7db-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:17.938178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:17.938177    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.040162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.040125    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c85f57d-183f-408c-a450-6d6ad119a7db-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.040347 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.040197    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.040347 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.040319    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.040459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.040377    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.040459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.040408    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.040591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.040516    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.040591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.040554    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.040703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.040644    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5ff\" (UniqueName: \"kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-kube-api-access-7r5ff\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.040807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.040791    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.041752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.041723    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.042214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.042183    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c85f57d-183f-408c-a450-6d6ad119a7db-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.042338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.042312    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.042905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.042879    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.043045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.043022    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.046977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.046945    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.049144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.049082    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.058373 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.058323    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.059570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.059547    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5ff\" (UniqueName: \"kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-kube-api-access-7r5ff\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.205677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.205647    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:18.220673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.220651    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"]
Nov 03 20:08:18.223180 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:08:18.223156    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce0df39_f9a8_4347_aa52_423d78661bfe.slice/crio-8238d9a753f4950c76b97efba740e3811ff872d02882ebfb0f680a0d5879efed WatchSource:0}: Error finding container 8238d9a753f4950c76b97efba740e3811ff872d02882ebfb0f680a0d5879efed: Status 404 returned error can't find the container with id 8238d9a753f4950c76b97efba740e3811ff872d02882ebfb0f680a0d5879efed
Nov 03 20:08:18.503510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.503472    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"]
Nov 03 20:08:18.503890 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:08:18.503862    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c85f57d_183f_408c_a450_6d6ad119a7db.slice/crio-b3a9f156abb0de4ebfb99c27ef9c355934960892e801641f5ba94699718b52c6 WatchSource:0}: Error finding container b3a9f156abb0de4ebfb99c27ef9c355934960892e801641f5ba94699718b52c6: Status 404 returned error can't find the container with id b3a9f156abb0de4ebfb99c27ef9c355934960892e801641f5ba94699718b52c6
Nov 03 20:08:18.506914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.506876    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:18.507000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.506983    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:18.507049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.507029    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:18.618875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:18.618781    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:19.230358 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:19.230324    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh" event={"ID":"6c85f57d-183f-408c-a450-6d6ad119a7db","Type":"ContainerStarted","Data":"887b1c1799d538caba2c3e7fe42b7234a959cb83671d40f8a5b5ea942aef2b09"}
Nov 03 20:08:19.230801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:19.230385    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh" event={"ID":"6c85f57d-183f-408c-a450-6d6ad119a7db","Type":"ContainerStarted","Data":"b3a9f156abb0de4ebfb99c27ef9c355934960892e801641f5ba94699718b52c6"}
Nov 03 20:08:19.249508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:19.234291    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd" event={"ID":"cce0df39-f9a8-4347-aa52-423d78661bfe","Type":"ContainerStarted","Data":"0db4f916868025d92718e603ee17744ee840eea09a4703873b4518a938cba925"}
Nov 03 20:08:19.249508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:19.234350    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd" event={"ID":"cce0df39-f9a8-4347-aa52-423d78661bfe","Type":"ContainerStarted","Data":"8238d9a753f4950c76b97efba740e3811ff872d02882ebfb0f680a0d5879efed"}
Nov 03 20:08:19.249508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:19.235123    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"
Nov 03 20:08:19.265978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:19.265922    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh" podStartSLOduration=2.265906259 podStartE2EDuration="2.265906259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:19.258126046 +0000 UTC m=+1851.413028493" watchObservedRunningTime="2026-11-03 20:08:19.265906259 +0000 UTC m=+1851.420808677"
Nov 03 20:08:19.300116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:19.300062    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd" podStartSLOduration=2.300048857 podStartE2EDuration="2.300048857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:19.297441479 +0000 UTC m=+1851.452343900" watchObservedRunningTime="2026-11-03 20:08:19.300048857 +0000 UTC m=+1851.454951275"
Nov 03 20:08:19.365508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:19.350921    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:20.155091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:20.155032    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:20.206607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:20.206571    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:20.208679 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:20.208647    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:08:20.208825 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:20.208734    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh" podUID="6c85f57d-183f-408c-a450-6d6ad119a7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 20:08:21.077676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:21.077636    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:21.206740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:21.206705    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:08:21.206958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:21.206764    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh" podUID="6c85f57d-183f-408c-a450-6d6ad119a7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 20:08:22.027386 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:22.027352    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:22.206476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:22.206441    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:08:22.206892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:22.206534    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh" podUID="6c85f57d-183f-408c-a450-6d6ad119a7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 20:08:22.763062 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:22.763011    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:23.206398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:23.206370    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:08:23.206579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:23.206443    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh" podUID="6c85f57d-183f-408c-a450-6d6ad119a7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 20:08:23.553030 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:23.552938    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:24.206494 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:24.206458    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:08:24.206678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:24.206537    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh" podUID="6c85f57d-183f-408c-a450-6d6ad119a7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 20:08:24.352012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:24.351976    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:25.115208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:25.115171    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:25.210234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:25.210195    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:25.210723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:25.210513    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:25.211239 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:25.211218    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:08:25.930245 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:25.930206    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:26.736824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:26.736789    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:27.442333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:27.442275    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:28.006309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:28.006274    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:28.433575 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:28.433369    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:28.905138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:28.905105    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:29.296697 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.296658    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"]
Nov 03 20:08:29.310763 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.310730    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"]
Nov 03 20:08:29.312363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.311060    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.317460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.316655    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-eb69cb-serving-cert\""
Nov 03 20:08:29.372564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.372473    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:29.436519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.415751    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.443666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.443623    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fff012-3266-4752-b836-fe60b7e4fb2b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.444429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.444060    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-metrics-certs\") pod \"maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.444857 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.444809    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh6gz\" (UniqueName: \"kubernetes.io/projected/d6fff012-3266-4752-b836-fe60b7e4fb2b-kube-api-access-gh6gz\") pod \"maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.502134 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.502100    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:29.546016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.545982    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.546200 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.546038    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fff012-3266-4752-b836-fe60b7e4fb2b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.546200 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.546139    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-metrics-certs\") pod \"maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.546335 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.546248    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh6gz\" (UniqueName: \"kubernetes.io/projected/d6fff012-3266-4752-b836-fe60b7e4fb2b-kube-api-access-gh6gz\") pod \"maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.547030 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.546965    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fff012-3266-4752-b836-fe60b7e4fb2b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.548882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.548860    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-metrics-certs\") pod \"maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.550431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.550406    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.558236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.558214    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh6gz\" (UniqueName: \"kubernetes.io/projected/d6fff012-3266-4752-b836-fe60b7e4fb2b-kube-api-access-gh6gz\") pod \"maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.631438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.631403    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:29.643209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.643155    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:29.928807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:29.928723    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"]
Nov 03 20:08:30.373747 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:30.373708    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf" event={"ID":"d6fff012-3266-4752-b836-fe60b7e4fb2b","Type":"ContainerStarted","Data":"4da13a8e47ba58a9bd57f1b317c9cfea36fd1aa3cd63176b980a39014a1d567d"}
Nov 03 20:08:30.410351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:30.410319    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:30.560504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:30.560182    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:31.052057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:31.051963    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:31.367410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:31.367327    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"
Nov 03 20:08:31.398332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:31.397123    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf" event={"ID":"d6fff012-3266-4752-b836-fe60b7e4fb2b","Type":"ContainerStarted","Data":"9637031f775870bfaa82d63e87db2137c387eed43656ee37b5f53d718f84c756"}
Nov 03 20:08:31.398332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:31.397562    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:31.434583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:31.434523    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf" podStartSLOduration=1.749506491 podStartE2EDuration="2.43447665s" totalImagesPullingTime="684.970159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:31.428434849 +0000 UTC m=+1863.583337284" watchObservedRunningTime="2026-11-03 20:08:31.43447665 +0000 UTC m=+1863.589379071"
Nov 03 20:08:31.488839 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:31.488752    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:31.622832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:31.622691    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:32.235919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:32.235884    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:33.136683 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:33.136636    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:33.659512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:33.657211    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"]
Nov 03 20:08:33.659512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:33.657915    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy" containerID="cri-o://0306132eca4a1e60a4c3e87c7710961024013605bb30c4ff5f816c8fbcbce15b" gracePeriod=30
Nov 03 20:08:34.435038 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:34.434990    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:35.129704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:35.129674    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:35.826651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:35.826617    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:36.635774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:36.635742    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:37.046232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:37.046196    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:37.412258 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:37.412171    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:08:37.441984 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:37.441948    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:38.254123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:38.254089    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:38.950401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:38.950374    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 20:08:39.080298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.080267    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-kube-api-access-d5w27\" (UniqueName: \"kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-kube-api-access-d5w27\") pod \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") "
Nov 03 20:08:39.080475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.080431    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-socket\") pod \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") "
Nov 03 20:08:39.080475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.080465    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istiod-ca-cert\") pod \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") "
Nov 03 20:08:39.080618 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.080517    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-certs\") pod \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") "
Nov 03 20:08:39.080618 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.080555    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-token\" (UniqueName: \"kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-token\") pod \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") "
Nov 03 20:08:39.080730 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.080667    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-podinfo\") pod \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") "
Nov 03 20:08:39.080817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.080799    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-data\") pod \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") "
Nov 03 20:08:39.080887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.080877    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-credential-socket\") pod \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") "
Nov 03 20:08:39.080940 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.080908    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-envoy\") pod \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\" (UID: \"129f5fbf-e7a2-436a-8a9f-1b52e42195f9\") "
Nov 03 20:08:39.081161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.080798    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-socket" pod "129f5fbf-e7a2-436a-8a9f-1b52e42195f9" (UID: "129f5fbf-e7a2-436a-8a9f-1b52e42195f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:39.081161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.080923    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istiod-ca-cert" pod "129f5fbf-e7a2-436a-8a9f-1b52e42195f9" (UID: "129f5fbf-e7a2-436a-8a9f-1b52e42195f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:39.081161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.081028    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-data" pod "129f5fbf-e7a2-436a-8a9f-1b52e42195f9" (UID: "129f5fbf-e7a2-436a-8a9f-1b52e42195f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:39.081389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.081200    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-credential-socket" pod "129f5fbf-e7a2-436a-8a9f-1b52e42195f9" (UID: "129f5fbf-e7a2-436a-8a9f-1b52e42195f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:39.081389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.081340    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-certs" pod "129f5fbf-e7a2-436a-8a9f-1b52e42195f9" (UID: "129f5fbf-e7a2-436a-8a9f-1b52e42195f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:39.081882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.081859    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:39.082012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.081996    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:39.082116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.082104    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:39.082207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.082196    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:39.082290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.082278    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:39.083285 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.083261    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-kube-api-access-d5w27" pod "129f5fbf-e7a2-436a-8a9f-1b52e42195f9" (UID: "129f5fbf-e7a2-436a-8a9f-1b52e42195f9"). InnerVolumeSpecName "kube-api-access-d5w27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:39.083405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.083326    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-podinfo" pod "129f5fbf-e7a2-436a-8a9f-1b52e42195f9" (UID: "129f5fbf-e7a2-436a-8a9f-1b52e42195f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:08:39.083560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.083530    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-envoy" pod "129f5fbf-e7a2-436a-8a9f-1b52e42195f9" (UID: "129f5fbf-e7a2-436a-8a9f-1b52e42195f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:39.083655 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.083636    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-token" pod "129f5fbf-e7a2-436a-8a9f-1b52e42195f9" (UID: "129f5fbf-e7a2-436a-8a9f-1b52e42195f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:39.184389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.184295    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:39.184389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.184327    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:39.184389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.184337    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:39.184389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.184347    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5w27\" (UniqueName: \"kubernetes.io/projected/129f5fbf-e7a2-436a-8a9f-1b52e42195f9-kube-api-access-d5w27\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:39.314275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.314234    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:39.535213 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.535174    5659 generic.go:356] "Generic (PLEG): container finished" podID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerID="0306132eca4a1e60a4c3e87c7710961024013605bb30c4ff5f816c8fbcbce15b" exitCode=0
Nov 03 20:08:39.535520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.535229    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"
Nov 03 20:08:39.535520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.535229    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" event={"ID":"129f5fbf-e7a2-436a-8a9f-1b52e42195f9","Type":"ContainerDied","Data":"0306132eca4a1e60a4c3e87c7710961024013605bb30c4ff5f816c8fbcbce15b"}
Nov 03 20:08:39.535520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.535298    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm" event={"ID":"129f5fbf-e7a2-436a-8a9f-1b52e42195f9","Type":"ContainerDied","Data":"e9553064512ba82d31adfba800fa77d7edeb186e77605b20304b94ad9b432d64"}
Nov 03 20:08:39.535520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.535335    5659 scope.go:122] "RemoveContainer" containerID="0306132eca4a1e60a4c3e87c7710961024013605bb30c4ff5f816c8fbcbce15b"
Nov 03 20:08:39.549903 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.549876    5659 scope.go:122] "RemoveContainer" containerID="0306132eca4a1e60a4c3e87c7710961024013605bb30c4ff5f816c8fbcbce15b"
Nov 03 20:08:39.550242 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:08:39.550225    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0306132eca4a1e60a4c3e87c7710961024013605bb30c4ff5f816c8fbcbce15b\": container with ID starting with 0306132eca4a1e60a4c3e87c7710961024013605bb30c4ff5f816c8fbcbce15b not found: ID does not exist" containerID="0306132eca4a1e60a4c3e87c7710961024013605bb30c4ff5f816c8fbcbce15b"
Nov 03 20:08:39.550303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.550253    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0306132eca4a1e60a4c3e87c7710961024013605bb30c4ff5f816c8fbcbce15b"} err="failed to get container status \"0306132eca4a1e60a4c3e87c7710961024013605bb30c4ff5f816c8fbcbce15b\": rpc error: code = NotFound desc = could not find container \"0306132eca4a1e60a4c3e87c7710961024013605bb30c4ff5f816c8fbcbce15b\": container with ID starting with 0306132eca4a1e60a4c3e87c7710961024013605bb30c4ff5f816c8fbcbce15b not found: ID does not exist"
Nov 03 20:08:39.563094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.563067    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"]
Nov 03 20:08:39.572148 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.572117    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-2djvm"]
Nov 03 20:08:39.853774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.853690    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"]
Nov 03 20:08:39.925706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:39.925671    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:40.064529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:40.050628    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" path="/var/lib/kubelet/pods/129f5fbf-e7a2-436a-8a9f-1b52e42195f9/volumes"
Nov 03 20:08:40.366711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:40.366452    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:41.138558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:41.138474    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:41.524576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:41.524545    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"]
Nov 03 20:08:41.525125 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:41.524785    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq" podUID="8f648e3d-ef7b-4282-a072-503ecb400a96" containerName="istio-proxy" containerID="cri-o://f9c21bad43da7b994095ee5ebc0aa43e7b3956af6c723fffca4133335038a2fc" gracePeriod=30
Nov 03 20:08:41.966991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:41.966950    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:42.734686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:42.734560    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:43.316406 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:43.316353    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:43.526514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:43.526397    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:44.523534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:44.523501    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:44.778036 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:44.777950    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:44.938579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:44.938544    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:45.227743 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:08:45.227687    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:08:45.779694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:45.779660    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:46.458446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:46.458414    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:46.602127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:46.602091    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:46.888550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:46.888244    5659 generic.go:356] "Generic (PLEG): container finished" podID="8f648e3d-ef7b-4282-a072-503ecb400a96" containerID="f9c21bad43da7b994095ee5ebc0aa43e7b3956af6c723fffca4133335038a2fc" exitCode=0
Nov 03 20:08:46.888550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:46.888444    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq" event={"ID":"8f648e3d-ef7b-4282-a072-503ecb400a96","Type":"ContainerDied","Data":"f9c21bad43da7b994095ee5ebc0aa43e7b3956af6c723fffca4133335038a2fc"}
Nov 03 20:08:46.890055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:46.889985    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:47.078454 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.078422    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 20:08:47.172997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.172943    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-socket\") pod \"8f648e3d-ef7b-4282-a072-503ecb400a96\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") "
Nov 03 20:08:47.173201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.173017    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-envoy\") pod \"8f648e3d-ef7b-4282-a072-503ecb400a96\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") "
Nov 03 20:08:47.173201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.173048    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-certs\") pod \"8f648e3d-ef7b-4282-a072-503ecb400a96\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") "
Nov 03 20:08:47.173201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.173078    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-credential-socket\") pod \"8f648e3d-ef7b-4282-a072-503ecb400a96\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") "
Nov 03 20:08:47.173201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.173149    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-data\") pod \"8f648e3d-ef7b-4282-a072-503ecb400a96\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") "
Nov 03 20:08:47.173201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.173203    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-token\" (UniqueName: \"kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-token\") pod \"8f648e3d-ef7b-4282-a072-503ecb400a96\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") "
Nov 03 20:08:47.173559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.173243    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f648e3d-ef7b-4282-a072-503ecb400a96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f648e3d-ef7b-4282-a072-503ecb400a96-istiod-ca-cert\") pod \"8f648e3d-ef7b-4282-a072-503ecb400a96\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") "
Nov 03 20:08:47.173559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.173419    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-kube-api-access-khztw\" (UniqueName: \"kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-kube-api-access-khztw\") pod \"8f648e3d-ef7b-4282-a072-503ecb400a96\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") "
Nov 03 20:08:47.174611 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.174580    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-credential-socket" pod "8f648e3d-ef7b-4282-a072-503ecb400a96" (UID: "8f648e3d-ef7b-4282-a072-503ecb400a96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:47.174839 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.174744    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-certs" pod "8f648e3d-ef7b-4282-a072-503ecb400a96" (UID: "8f648e3d-ef7b-4282-a072-503ecb400a96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:47.174950 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.174934    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-podinfo\") pod \"8f648e3d-ef7b-4282-a072-503ecb400a96\" (UID: \"8f648e3d-ef7b-4282-a072-503ecb400a96\") "
Nov 03 20:08:47.177281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.175162    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f648e3d-ef7b-4282-a072-503ecb400a96-istiod-ca-cert" pod "8f648e3d-ef7b-4282-a072-503ecb400a96" (UID: "8f648e3d-ef7b-4282-a072-503ecb400a96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:47.177281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.175240    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-socket" pod "8f648e3d-ef7b-4282-a072-503ecb400a96" (UID: "8f648e3d-ef7b-4282-a072-503ecb400a96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:47.178068 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.177889    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:47.178068 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.178016    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:47.178068 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.178033    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:47.178068 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.178049    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f648e3d-ef7b-4282-a072-503ecb400a96-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:47.178386 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.178359    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-data" pod "8f648e3d-ef7b-4282-a072-503ecb400a96" (UID: "8f648e3d-ef7b-4282-a072-503ecb400a96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:47.181686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.181657    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-envoy" pod "8f648e3d-ef7b-4282-a072-503ecb400a96" (UID: "8f648e3d-ef7b-4282-a072-503ecb400a96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:47.189895 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.189871    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-token" pod "8f648e3d-ef7b-4282-a072-503ecb400a96" (UID: "8f648e3d-ef7b-4282-a072-503ecb400a96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:47.189989 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.189910    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-kube-api-access-khztw" pod "8f648e3d-ef7b-4282-a072-503ecb400a96" (UID: "8f648e3d-ef7b-4282-a072-503ecb400a96"). InnerVolumeSpecName "kube-api-access-khztw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:47.191299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.191257    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-podinfo" pod "8f648e3d-ef7b-4282-a072-503ecb400a96" (UID: "8f648e3d-ef7b-4282-a072-503ecb400a96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:08:47.201232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.201208    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:47.282459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.280465    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:47.282459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.280535    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:47.282459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.280551    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:47.282459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.280569    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khztw\" (UniqueName: \"kubernetes.io/projected/8f648e3d-ef7b-4282-a072-503ecb400a96-kube-api-access-khztw\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:47.282459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.280586    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f648e3d-ef7b-4282-a072-503ecb400a96-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:47.634506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.627967    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:47.791505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.785931    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:47.909972 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.909532    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"]
Nov 03 20:08:47.912586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.910559    5659 cpu_manager.go:414] "containerMap: removing container" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy"
Nov 03 20:08:47.912586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.910583    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy"
Nov 03 20:08:47.912586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.910598    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8f648e3d-ef7b-4282-a072-503ecb400a96" containerName="istio-proxy"
Nov 03 20:08:47.912586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.910604    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f648e3d-ef7b-4282-a072-503ecb400a96" containerName="istio-proxy"
Nov 03 20:08:47.912586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.910761    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="129f5fbf-e7a2-436a-8a9f-1b52e42195f9" containerName="istio-proxy"
Nov 03 20:08:47.912586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.910772    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f648e3d-ef7b-4282-a072-503ecb400a96" containerName="istio-proxy"
Nov 03 20:08:47.925274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.923144    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:47.927829 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.927742    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"]
Nov 03 20:08:47.931921 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.931893    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-88a18e-openshift-default-dockercfg-bs4qg\""
Nov 03 20:08:47.947107 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.946120    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:47.958586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.958549    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq" event={"ID":"8f648e3d-ef7b-4282-a072-503ecb400a96","Type":"ContainerDied","Data":"f1a1cc3a85794b02f1a8018cf2eef00733594ee73e099e98811d7a6e6678b775"}
Nov 03 20:08:47.958737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.958629    5659 scope.go:122] "RemoveContainer" containerID="f9c21bad43da7b994095ee5ebc0aa43e7b3956af6c723fffca4133335038a2fc"
Nov 03 20:08:47.958833 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:47.958816    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"
Nov 03 20:08:48.016755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.016720    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85qwm\" (UniqueName: \"kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-kube-api-access-85qwm\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.017086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.017066    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-credential-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.017311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.017293    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-envoy\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.017435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.017420    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-token\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.017598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.017580    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-data\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.017855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.017836    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-certs\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.018074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.018053    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istiod-ca-cert\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.019469 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.019446    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-podinfo\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.019647 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.019630    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.073250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.073217    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"]
Nov 03 20:08:48.078688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.078654    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-lm9zq"]
Nov 03 20:08:48.095057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.095026    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:48.132624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.132553    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-envoy\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.134931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.134904    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-token\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.135463 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.135440    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-data\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.136067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.136046    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-certs\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.136549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.136531    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istiod-ca-cert\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.148570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.136441    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-certs\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.148570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.135980    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-data\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.148570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.137299    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istiod-ca-cert\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.148570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.143331    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-podinfo\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.148570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.143371    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.148570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.143444    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85qwm\" (UniqueName: \"kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-kube-api-access-85qwm\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.148570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.143703    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-credential-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.148570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.144080    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-credential-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.148570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.145154    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-socket\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.148570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.147178    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-envoy\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.149567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.149538    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-podinfo\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.149840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.149820    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-token\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.163561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.158338    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85qwm\" (UniqueName: \"kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-kube-api-access-85qwm\") pod \"e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") " pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.250506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.250459    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:48.568561 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:08:48.567957    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a6ec71_2a2b_4e6c_aab0_56512872cb10.slice/crio-069bf541c028c57a8889938e538fa8ba37223918e830a1bfba9099a803b7c61a WatchSource:0}: Error finding container 069bf541c028c57a8889938e538fa8ba37223918e830a1bfba9099a803b7c61a: Status 404 returned error can't find the container with id 069bf541c028c57a8889938e538fa8ba37223918e830a1bfba9099a803b7c61a
Nov 03 20:08:48.571251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.571207    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:48.571352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.571305    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:48.571397 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.571356    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:48.572002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.571952    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"]
Nov 03 20:08:48.932710 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:48.932673    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:49.015504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:49.011165    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx" event={"ID":"87a6ec71-2a2b-4e6c-aab0-56512872cb10","Type":"ContainerStarted","Data":"bcfc82f4b840049384bd666a9dbc6a89c28fcf4be94d55d6f55976ec189d32d4"}
Nov 03 20:08:49.015504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:49.011239    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx" event={"ID":"87a6ec71-2a2b-4e6c-aab0-56512872cb10","Type":"ContainerStarted","Data":"069bf541c028c57a8889938e538fa8ba37223918e830a1bfba9099a803b7c61a"}
Nov 03 20:08:49.054064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:49.054006    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx" podStartSLOduration=2.05398835 podStartE2EDuration="2.05398835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:49.037398612 +0000 UTC m=+1881.192301032" watchObservedRunningTime="2026-11-03 20:08:49.05398835 +0000 UTC m=+1881.208890769"
Nov 03 20:08:49.255124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:49.252507    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:49.271521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:49.269558    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:08:49.271521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:49.269783    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx" podUID="87a6ec71-2a2b-4e6c-aab0-56512872cb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:08:49.713445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:49.713407    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:49.974313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:49.974234    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f648e3d-ef7b-4282-a072-503ecb400a96" path="/var/lib/kubelet/pods/8f648e3d-ef7b-4282-a072-503ecb400a96/volumes"
Nov 03 20:08:50.262644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:50.256908    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:08:50.262844 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:50.262651    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx" podUID="87a6ec71-2a2b-4e6c-aab0-56512872cb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:08:50.560568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:50.553327    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:51.251001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:51.250965    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:08:51.251610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:51.251560    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx" podUID="87a6ec71-2a2b-4e6c-aab0-56512872cb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:08:51.481408 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:51.481337    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:51.626226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:51.626141    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:52.246504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:52.242869    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:52.254504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:52.251116    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:08:52.254504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:52.251210    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx" podUID="87a6ec71-2a2b-4e6c-aab0-56512872cb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:08:52.923307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:52.923276    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:53.077169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:53.077135    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:53.269505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:53.265253    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:08:53.269505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:53.265313    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx" podUID="87a6ec71-2a2b-4e6c-aab0-56512872cb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:08:53.486836 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:53.486801    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:54.252150 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:54.252112    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:08:54.252460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:54.252430    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx" podUID="87a6ec71-2a2b-4e6c-aab0-56512872cb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:08:54.330090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:54.330042    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:54.907765 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:54.907725    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5" podUID="81b2ae07-b323-4ad4-99fc-45d5ad248aa0" containerName="main" containerID="cri-o://0d0f879d32e986e8c9a16cf250d7634df1362c5008ee2c6c26995b774069d1ad" gracePeriod=45
Nov 03 20:08:55.054506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.048968    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:55.079172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.075842    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5" podUID="81b2ae07-b323-4ad4-99fc-45d5ad248aa0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": dial tcp 10.133.0.109:8000: connect: connection refused"
Nov 03 20:08:55.155726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.155690    5659 generic.go:356] "Generic (PLEG): container finished" podID="81b2ae07-b323-4ad4-99fc-45d5ad248aa0" containerID="0d0f879d32e986e8c9a16cf250d7634df1362c5008ee2c6c26995b774069d1ad" exitCode=0
Nov 03 20:08:55.156016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.155994    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5" event={"ID":"81b2ae07-b323-4ad4-99fc-45d5ad248aa0","Type":"ContainerDied","Data":"0d0f879d32e986e8c9a16cf250d7634df1362c5008ee2c6c26995b774069d1ad"}
Nov 03 20:08:55.270300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.270264    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:08:55.275115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.270543    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx" podUID="87a6ec71-2a2b-4e6c-aab0-56512872cb10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:08:55.321109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.321075    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:08:55.412950 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.412913    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-model-cache\") pod \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") "
Nov 03 20:08:55.413459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.413028    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-home\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-home\") pod \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") "
Nov 03 20:08:55.413459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.413074    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tmp-dir\") pod \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") "
Nov 03 20:08:55.413459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.413119    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-kube-api-access-g7ldn\" (UniqueName: \"kubernetes.io/projected/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-kube-api-access-g7ldn\") pod \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") "
Nov 03 20:08:55.413459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.413173    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tls-certs\" (UniqueName: \"kubernetes.io/secret/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tls-certs\") pod \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") "
Nov 03 20:08:55.413459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.413229    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-model-cache" pod "81b2ae07-b323-4ad4-99fc-45d5ad248aa0" (UID: "81b2ae07-b323-4ad4-99fc-45d5ad248aa0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:55.413459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.413242    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-dshm\") pod \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\" (UID: \"81b2ae07-b323-4ad4-99fc-45d5ad248aa0\") "
Nov 03 20:08:55.413459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.413284    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-home" pod "81b2ae07-b323-4ad4-99fc-45d5ad248aa0" (UID: "81b2ae07-b323-4ad4-99fc-45d5ad248aa0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:55.413459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.413381    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tmp-dir" pod "81b2ae07-b323-4ad4-99fc-45d5ad248aa0" (UID: "81b2ae07-b323-4ad4-99fc-45d5ad248aa0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:55.414095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.414066    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:55.414147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.414095    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:55.414147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.414109    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:55.415516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.415494    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-kube-api-access-g7ldn" pod "81b2ae07-b323-4ad4-99fc-45d5ad248aa0" (UID: "81b2ae07-b323-4ad4-99fc-45d5ad248aa0"). InnerVolumeSpecName "kube-api-access-g7ldn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:55.415748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.415729    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tls-certs" pod "81b2ae07-b323-4ad4-99fc-45d5ad248aa0" (UID: "81b2ae07-b323-4ad4-99fc-45d5ad248aa0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:55.415800 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.415738    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-dshm" pod "81b2ae07-b323-4ad4-99fc-45d5ad248aa0" (UID: "81b2ae07-b323-4ad4-99fc-45d5ad248aa0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:55.515579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.515544    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7ldn\" (UniqueName: \"kubernetes.io/projected/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-kube-api-access-g7ldn\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:55.515579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.515573    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:55.515579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.515583    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b2ae07-b323-4ad4-99fc-45d5ad248aa0-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:55.730985 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:55.730949    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:56.167551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:56.167526    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"
Nov 03 20:08:56.167714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:56.167518    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5" event={"ID":"81b2ae07-b323-4ad4-99fc-45d5ad248aa0","Type":"ContainerDied","Data":"267591add7376b55cf0b22eec38c52804887c67dcb7967bfe0afdbc440e23db2"}
Nov 03 20:08:56.167714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:56.167636    5659 scope.go:122] "RemoveContainer" containerID="0d0f879d32e986e8c9a16cf250d7634df1362c5008ee2c6c26995b774069d1ad"
Nov 03 20:08:56.187002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:56.186970    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"]
Nov 03 20:08:56.190507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:56.190471    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bfc240/auth-test-model-bfc240-kserve-6987dbf9b9-247h5"]
Nov 03 20:08:56.256269 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:56.256189    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:56.256714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:56.256693    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:56.257631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:56.257608    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:08:56.365896 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:56.365868    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:57.261665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:57.261626    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:58.005504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:58.005349    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b2ae07-b323-4ad4-99fc-45d5ad248aa0" path="/var/lib/kubelet/pods/81b2ae07-b323-4ad4-99fc-45d5ad248aa0/volumes"
Nov 03 20:08:58.435082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:58.434992    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:58.915557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:58.915475    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:59.059704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:59.059671    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:59.571329 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:08:59.571286    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:00.137253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:00.137218    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:01.157916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.157885    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"]
Nov 03 20:09:01.158530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.158514    5659 cpu_manager.go:414] "containerMap: removing container" podUID="81b2ae07-b323-4ad4-99fc-45d5ad248aa0" containerName="main"
Nov 03 20:09:01.158586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.158531    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b2ae07-b323-4ad4-99fc-45d5ad248aa0" containerName="main"
Nov 03 20:09:01.158646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.158637    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b2ae07-b323-4ad4-99fc-45d5ad248aa0" containerName="main"
Nov 03 20:09:01.163931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.163910    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.174898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.174876    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a0d4ec-serving-cert\""
Nov 03 20:09:01.177586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.177551    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"]
Nov 03 20:09:01.215686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.215630    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:01.316823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.316754    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aca7ab0-c961-4029-9d38-5e5697f39f76-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.317474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.317228    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb2qg\" (UniqueName: \"kubernetes.io/projected/8aca7ab0-c961-4029-9d38-5e5697f39f76-kube-api-access-qb2qg\") pod \"maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.317768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.317736    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-metrics-certs\") pod \"maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.318093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.318073    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.355371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.355332    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:01.419647 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.419552    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aca7ab0-c961-4029-9d38-5e5697f39f76-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.419647 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.419604    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb2qg\" (UniqueName: \"kubernetes.io/projected/8aca7ab0-c961-4029-9d38-5e5697f39f76-kube-api-access-qb2qg\") pod \"maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.419969 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.419929    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-metrics-certs\") pod \"maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.420138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.420123    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.420320 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.420298    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aca7ab0-c961-4029-9d38-5e5697f39f76-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.423210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.423186    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-metrics-certs\") pod \"maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.423300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.423192    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.435459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.435429    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb2qg\" (UniqueName: \"kubernetes.io/projected/8aca7ab0-c961-4029-9d38-5e5697f39f76-kube-api-access-qb2qg\") pod \"maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.479217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.479181    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:01.512677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.512648    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:01.824844 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:01.824810    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"]
Nov 03 20:09:01.825644 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:09:01.825616    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aca7ab0_c961_4029_9d38_5e5697f39f76.slice/crio-c73daa4a6ac38ab30cda44869a103e38c61df0c6004d68830a1a7939dc111fa0 WatchSource:0}: Error finding container c73daa4a6ac38ab30cda44869a103e38c61df0c6004d68830a1a7939dc111fa0: Status 404 returned error can't find the container with id c73daa4a6ac38ab30cda44869a103e38c61df0c6004d68830a1a7939dc111fa0
Nov 03 20:09:02.190864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:02.190789    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:02.262101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:02.262063    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f" event={"ID":"8aca7ab0-c961-4029-9d38-5e5697f39f76","Type":"ContainerStarted","Data":"c73daa4a6ac38ab30cda44869a103e38c61df0c6004d68830a1a7939dc111fa0"}
Nov 03 20:09:02.334803 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:02.334760    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:02.688671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:02.688641    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:02.832922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:02.832892    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:02.873924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:02.873878    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"]
Nov 03 20:09:02.878553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:02.878525    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:02.898553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:02.898527    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"]
Nov 03 20:09:02.985873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:02.985840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:03.029244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.029208    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.029394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.029255    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6671998-ae34-4319-8e68-9785a7d6558d-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.029394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.029341    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-446tm\" (UniqueName: \"kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-kube-api-access-446tm\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.029524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.029436    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6671998-ae34-4319-8e68-9785a7d6558d-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.029524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.029477    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.029610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.029564    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.029610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.029584    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.029716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.029693    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.029749 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.029730    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.131086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.131053    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6671998-ae34-4319-8e68-9785a7d6558d-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.131289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.131112    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.131289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.131159    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.131289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.131174    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.131289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.131220    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.131548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.131292    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.131548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.131471    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.131664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.131535    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.131664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.131546    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6671998-ae34-4319-8e68-9785a7d6558d-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.131664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.131615    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.131664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.131628    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-446tm\" (UniqueName: \"kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-kube-api-access-446tm\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.131852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.131773    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.131904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.131873    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6671998-ae34-4319-8e68-9785a7d6558d-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.132088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.132063    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.133680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.133655    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.134521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.134445    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6671998-ae34-4319-8e68-9785a7d6558d-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.145578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.145066    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.145578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.145298    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-446tm\" (UniqueName: \"kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-kube-api-access-446tm\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.194425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.194391    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:03.281532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.280017    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f" event={"ID":"8aca7ab0-c961-4029-9d38-5e5697f39f76","Type":"ContainerStarted","Data":"bfa4b3cdfc4796a833d51fb56de4a50c977da89c403d2ccf8e27d77f89597ebb"}
Nov 03 20:09:03.281532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.281085    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:03.303991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.303942    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f" podStartSLOduration=1.829226507 podStartE2EDuration="2.303927085s" totalImagesPullingTime="474.700578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:03.300920445 +0000 UTC m=+1895.455822867" watchObservedRunningTime="2026-11-03 20:09:03.303927085 +0000 UTC m=+1895.458829504"
Nov 03 20:09:03.550498 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.550453    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"]
Nov 03 20:09:03.551401 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:09:03.551377    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6671998_ae34_4319_8e68_9785a7d6558d.slice/crio-997b1618835a33607675887fc328d3443cd572120e8d573f8ed5f7d6b4704ab1 WatchSource:0}: Error finding container 997b1618835a33607675887fc328d3443cd572120e8d573f8ed5f7d6b4704ab1: Status 404 returned error can't find the container with id 997b1618835a33607675887fc328d3443cd572120e8d573f8ed5f7d6b4704ab1
Nov 03 20:09:03.554207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.554163    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:03.554313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.554251    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:03.554313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.554296    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:03.964946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:03.964916    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:04.294047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.293959    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz" event={"ID":"a6671998-ae34-4319-8e68-9785a7d6558d","Type":"ContainerStarted","Data":"dd6324defaae3f0eceddaca8259fba2129d9868bf099d10ee180296fb93b5e1a"}
Nov 03 20:09:04.295523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.295496    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz" event={"ID":"a6671998-ae34-4319-8e68-9785a7d6558d","Type":"ContainerStarted","Data":"997b1618835a33607675887fc328d3443cd572120e8d573f8ed5f7d6b4704ab1"}
Nov 03 20:09:04.326718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.326665    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz" podStartSLOduration=2.326649577 podStartE2EDuration="2.326649577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:04.321591317 +0000 UTC m=+1896.476493737" watchObservedRunningTime="2026-11-03 20:09:04.326649577 +0000 UTC m=+1896.481551996"
Nov 03 20:09:04.361297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.361255    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:04.496506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.496431    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"]
Nov 03 20:09:04.507137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.507012    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:04.510333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.509210    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"]
Nov 03 20:09:04.521060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.521031    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"]
Nov 03 20:09:04.521346 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.521327    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"
Nov 03 20:09:04.531718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.530799    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-46a6d7\""
Nov 03 20:09:04.531718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.531580    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-46a6d7-dockercfg-jklpd\""
Nov 03 20:09:04.596708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.569624    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbpkj\" (UniqueName: \"kubernetes.io/projected/7c195f09-b9ef-45a0-89d6-07ea2a203277-kube-api-access-sbpkj\") pod \"payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229\" (UID: \"7c195f09-b9ef-45a0-89d6-07ea2a203277\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"
Nov 03 20:09:04.596708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.569879    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c195f09-b9ef-45a0-89d6-07ea2a203277-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229\" (UID: \"7c195f09-b9ef-45a0-89d6-07ea2a203277\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"
Nov 03 20:09:04.612522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.607529    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"]
Nov 03 20:09:04.981506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.956709    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"
Nov 03 20:09:04.981506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.962004    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c195f09-b9ef-45a0-89d6-07ea2a203277-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229\" (UID: \"7c195f09-b9ef-45a0-89d6-07ea2a203277\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"
Nov 03 20:09:04.981506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.962240    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbpkj\" (UniqueName: \"kubernetes.io/projected/7c195f09-b9ef-45a0-89d6-07ea2a203277-kube-api-access-sbpkj\") pod \"payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229\" (UID: \"7c195f09-b9ef-45a0-89d6-07ea2a203277\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"
Nov 03 20:09:04.981506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.963643    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c195f09-b9ef-45a0-89d6-07ea2a203277-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229\" (UID: \"7c195f09-b9ef-45a0-89d6-07ea2a203277\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"
Nov 03 20:09:04.981506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:04.964405    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"]
Nov 03 20:09:05.013194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.013158    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbpkj\" (UniqueName: \"kubernetes.io/projected/7c195f09-b9ef-45a0-89d6-07ea2a203277-kube-api-access-sbpkj\") pod \"payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229\" (UID: \"7c195f09-b9ef-45a0-89d6-07ea2a203277\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"
Nov 03 20:09:05.073509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.072908    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c981a786-15c1-4282-9823-8a937781f2fe-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg\" (UID: \"c981a786-15c1-4282-9823-8a937781f2fe\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"
Nov 03 20:09:05.073509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.073202    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z78r\" (UniqueName: \"kubernetes.io/projected/c981a786-15c1-4282-9823-8a937781f2fe-kube-api-access-9z78r\") pod \"payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg\" (UID: \"c981a786-15c1-4282-9823-8a937781f2fe\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"
Nov 03 20:09:05.174986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.174947    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c981a786-15c1-4282-9823-8a937781f2fe-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg\" (UID: \"c981a786-15c1-4282-9823-8a937781f2fe\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"
Nov 03 20:09:05.175215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.175148    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z78r\" (UniqueName: \"kubernetes.io/projected/c981a786-15c1-4282-9823-8a937781f2fe-kube-api-access-9z78r\") pod \"payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg\" (UID: \"c981a786-15c1-4282-9823-8a937781f2fe\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"
Nov 03 20:09:05.175855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.175831    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c981a786-15c1-4282-9823-8a937781f2fe-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg\" (UID: \"c981a786-15c1-4282-9823-8a937781f2fe\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"
Nov 03 20:09:05.191549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.191523    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z78r\" (UniqueName: \"kubernetes.io/projected/c981a786-15c1-4282-9823-8a937781f2fe-kube-api-access-9z78r\") pod \"payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg\" (UID: \"c981a786-15c1-4282-9823-8a937781f2fe\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"
Nov 03 20:09:05.195339 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.195308    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:05.196936 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.196902    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 20:09:05.197060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.196973    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz" podUID="a6671998-ae34-4319-8e68-9785a7d6558d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 20:09:05.263556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.261790    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"
Nov 03 20:09:05.266634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.266610    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:05.354802 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.354763    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"
Nov 03 20:09:05.670538 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:09:05.670457    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c195f09_b9ef_45a0_89d6_07ea2a203277.slice/crio-3ccfd15dcb235eaa430b51b267de518f508c201343c794478a4f63019dc478f4 WatchSource:0}: Error finding container 3ccfd15dcb235eaa430b51b267de518f508c201343c794478a4f63019dc478f4: Status 404 returned error can't find the container with id 3ccfd15dcb235eaa430b51b267de518f508c201343c794478a4f63019dc478f4
Nov 03 20:09:05.672796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.672621    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"]
Nov 03 20:09:05.762505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:05.761759    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"]
Nov 03 20:09:06.194822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:06.194731    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 20:09:06.194822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:06.194800    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz" podUID="a6671998-ae34-4319-8e68-9785a7d6558d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 20:09:06.250058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:06.250023    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:06.368493 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:06.367077    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229" event={"ID":"7c195f09-b9ef-45a0-89d6-07ea2a203277","Type":"ContainerStarted","Data":"f90c901c67e88f92bff67effb2025c35dd61e7f28af42d1cf65da1825123dc68"}
Nov 03 20:09:06.368493 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:06.367148    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229" event={"ID":"7c195f09-b9ef-45a0-89d6-07ea2a203277","Type":"ContainerStarted","Data":"3ccfd15dcb235eaa430b51b267de518f508c201343c794478a4f63019dc478f4"}
Nov 03 20:09:06.368493 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:06.368340    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"
Nov 03 20:09:06.383579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:06.381263    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg" event={"ID":"c981a786-15c1-4282-9823-8a937781f2fe","Type":"ContainerStarted","Data":"d94f75e6653a403e2071874c0a6b1d2c080be663db7b0b4eeb44620ef7dd9a16"}
Nov 03 20:09:06.388369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:06.386792    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"
Nov 03 20:09:06.388678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:06.388643    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg" event={"ID":"c981a786-15c1-4282-9823-8a937781f2fe","Type":"ContainerStarted","Data":"31de108f82abca9f8b4035c65f02953538518eb36c4a0e740e3b0b059c792e3f"}
Nov 03 20:09:06.403516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:06.403138    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229" podStartSLOduration=2.403121566 podStartE2EDuration="2.403121566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:06.396877502 +0000 UTC m=+1898.551779939" watchObservedRunningTime="2026-11-03 20:09:06.403121566 +0000 UTC m=+1898.558023987"
Nov 03 20:09:06.450557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:06.430801    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg" podStartSLOduration=2.430781639 podStartE2EDuration="2.430781639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:06.428122863 +0000 UTC m=+1898.583025283" watchObservedRunningTime="2026-11-03 20:09:06.430781639 +0000 UTC m=+1898.585684058"
Nov 03 20:09:07.049509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:07.043776    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:07.195241 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:07.195202    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 20:09:07.195439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:07.195272    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz" podUID="a6671998-ae34-4319-8e68-9785a7d6558d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 20:09:07.872564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:07.872509    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:08.194965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:08.194885    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 20:09:08.194965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:08.194941    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz" podUID="a6671998-ae34-4319-8e68-9785a7d6558d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 20:09:08.657508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:08.656932    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:08.827558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:08.826914    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:09.194915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:09.194877    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 20:09:09.195555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:09.195520    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz" podUID="a6671998-ae34-4319-8e68-9785a7d6558d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 20:09:09.623197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:09.623163    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:10.195342 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:10.195302    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 20:09:10.195775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:10.195361    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz" podUID="a6671998-ae34-4319-8e68-9785a7d6558d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 20:09:10.312681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:10.312646    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:09:10.357906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:10.357786    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:10.490892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:10.490860    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:11.200331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:11.200301    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:11.200815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:11.200773    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:11.201598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:11.201578    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:09:11.411992 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:11.411935    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:11.424551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:11.419535    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"]
Nov 03 20:09:11.573037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:11.570279    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:11.921854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:11.921776    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:12.112198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:12.112163    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:12.402006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:12.401965    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:12.623363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:12.622340    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:12.769393 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:12.769354    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:12.901957 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:12.901921    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:13.552297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:13.552260    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:14.349601 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:14.349570    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:15.210596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:15.210562    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:15.253376 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:09:15.253336    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:09:16.197459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:16.197427    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"]
Nov 03 20:09:16.197683 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:16.197657    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy" containerID="cri-o://1e63db2684ed19d8085bb80b751e0fec207dff5efc771072776158159b245b15" gracePeriod=30
Nov 03 20:09:16.271230 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:16.271198    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:16.633319 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:16.633284    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:16.791200 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:16.791167    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:17.557441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.557407    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:17.620323 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.620285    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"]
Nov 03 20:09:17.628187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.628161    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"
Nov 03 20:09:17.636012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.635985    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"]
Nov 03 20:09:17.641746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.641719    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-98529d-dockercfg-rgksk\""
Nov 03 20:09:17.683505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.679354    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-98529d\""
Nov 03 20:09:17.824556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.772227    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7\" (UID: \"de7c4c4d-2e02-4c99-b9e0-8f15391d7871\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"
Nov 03 20:09:17.824556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.772281    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb86n\" (UniqueName: \"kubernetes.io/projected/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-kube-api-access-bb86n\") pod \"payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7\" (UID: \"de7c4c4d-2e02-4c99-b9e0-8f15391d7871\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"
Nov 03 20:09:17.832944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.832907    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:17.884506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.878567    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7\" (UID: \"de7c4c4d-2e02-4c99-b9e0-8f15391d7871\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"
Nov 03 20:09:17.884506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.878614    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb86n\" (UniqueName: \"kubernetes.io/projected/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-kube-api-access-bb86n\") pod \"payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7\" (UID: \"de7c4c4d-2e02-4c99-b9e0-8f15391d7871\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"
Nov 03 20:09:17.884506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.880439    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7\" (UID: \"de7c4c4d-2e02-4c99-b9e0-8f15391d7871\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"
Nov 03 20:09:17.911251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.906246    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb86n\" (UniqueName: \"kubernetes.io/projected/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-kube-api-access-bb86n\") pod \"payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7\" (UID: \"de7c4c4d-2e02-4c99-b9e0-8f15391d7871\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"
Nov 03 20:09:17.964680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:17.964641    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"
Nov 03 20:09:18.264819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:18.264788    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"]
Nov 03 20:09:18.265060 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:09:18.264881    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7c4c4d_2e02_4c99_b9e0_8f15391d7871.slice/crio-8dd7618243d63736eb5e1799ac284bc444d5fb9a5f7c9f0f45bc40f3f8e6026b WatchSource:0}: Error finding container 8dd7618243d63736eb5e1799ac284bc444d5fb9a5f7c9f0f45bc40f3f8e6026b: Status 404 returned error can't find the container with id 8dd7618243d63736eb5e1799ac284bc444d5fb9a5f7c9f0f45bc40f3f8e6026b
Nov 03 20:09:18.381842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:18.376804    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:18.410747 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:18.410713    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"
Nov 03 20:09:18.411208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:18.411187    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"
Nov 03 20:09:18.525470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:18.525387    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:18.674522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:18.674454    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:18.884599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:18.873978    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7" event={"ID":"de7c4c4d-2e02-4c99-b9e0-8f15391d7871","Type":"ContainerStarted","Data":"e371e3e73b5059592ffdc3fbc899049cb319e3e96b864bba71471a04e5931da7"}
Nov 03 20:09:18.884599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:18.874047    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7" event={"ID":"de7c4c4d-2e02-4c99-b9e0-8f15391d7871","Type":"ContainerStarted","Data":"8dd7618243d63736eb5e1799ac284bc444d5fb9a5f7c9f0f45bc40f3f8e6026b"}
Nov 03 20:09:18.884599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:18.875192    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"
Nov 03 20:09:18.899897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:18.899847    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7" podStartSLOduration=1.899833487 podStartE2EDuration="1.899833487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:18.896324286 +0000 UTC m=+1911.051226707" watchObservedRunningTime="2026-11-03 20:09:18.899833487 +0000 UTC m=+1911.054735904"
Nov 03 20:09:19.563287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:19.563236    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:19.971372 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:19.971328    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd" podUID="5d59ee3e-419f-4f34-b4f7-095287a8f640" containerName="main" containerID="cri-o://456ca519eadf85ef305435284d650284f37055ffb8ce30c7d3d84a974c576a33" gracePeriod=45
Nov 03 20:09:19.987046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:19.987014    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd" podUID="5d59ee3e-419f-4f34-b4f7-095287a8f640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": dial tcp 10.133.0.108:8000: connect: connection refused"
Nov 03 20:09:20.275891 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.275860    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:09:20.360835 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.360689    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:20.392564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.391028    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-dshm\") pod \"5d59ee3e-419f-4f34-b4f7-095287a8f640\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") "
Nov 03 20:09:20.392564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.391121    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-tmp-dir\") pod \"5d59ee3e-419f-4f34-b4f7-095287a8f640\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") "
Nov 03 20:09:20.392564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.391161    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d59ee3e-419f-4f34-b4f7-095287a8f640-kube-api-access-vgtt5\" (UniqueName: \"kubernetes.io/projected/5d59ee3e-419f-4f34-b4f7-095287a8f640-kube-api-access-vgtt5\") pod \"5d59ee3e-419f-4f34-b4f7-095287a8f640\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") "
Nov 03 20:09:20.392564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.391267    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-home\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-home\") pod \"5d59ee3e-419f-4f34-b4f7-095287a8f640\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") "
Nov 03 20:09:20.392564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.391308    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-model-cache\") pod \"5d59ee3e-419f-4f34-b4f7-095287a8f640\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") "
Nov 03 20:09:20.392564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.391416    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d59ee3e-419f-4f34-b4f7-095287a8f640-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d59ee3e-419f-4f34-b4f7-095287a8f640-tls-certs\") pod \"5d59ee3e-419f-4f34-b4f7-095287a8f640\" (UID: \"5d59ee3e-419f-4f34-b4f7-095287a8f640\") "
Nov 03 20:09:20.417507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.405959    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-home" pod "5d59ee3e-419f-4f34-b4f7-095287a8f640" (UID: "5d59ee3e-419f-4f34-b4f7-095287a8f640"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:20.417507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.406162    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-tmp-dir" pod "5d59ee3e-419f-4f34-b4f7-095287a8f640" (UID: "5d59ee3e-419f-4f34-b4f7-095287a8f640"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:20.417507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.406316    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-model-cache" pod "5d59ee3e-419f-4f34-b4f7-095287a8f640" (UID: "5d59ee3e-419f-4f34-b4f7-095287a8f640"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:20.447358 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.446950    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-dshm" pod "5d59ee3e-419f-4f34-b4f7-095287a8f640" (UID: "5d59ee3e-419f-4f34-b4f7-095287a8f640"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:20.451046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.451010    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d59ee3e-419f-4f34-b4f7-095287a8f640-tls-certs" pod "5d59ee3e-419f-4f34-b4f7-095287a8f640" (UID: "5d59ee3e-419f-4f34-b4f7-095287a8f640"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:09:20.452670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.452639    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d59ee3e-419f-4f34-b4f7-095287a8f640-kube-api-access-vgtt5" pod "5d59ee3e-419f-4f34-b4f7-095287a8f640" (UID: "5d59ee3e-419f-4f34-b4f7-095287a8f640"). InnerVolumeSpecName "kube-api-access-vgtt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:20.494117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.494040    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:20.494117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.494067    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgtt5\" (UniqueName: \"kubernetes.io/projected/5d59ee3e-419f-4f34-b4f7-095287a8f640-kube-api-access-vgtt5\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:20.494117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.494076    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:20.494117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.494084    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:20.494117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.494094    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d59ee3e-419f-4f34-b4f7-095287a8f640-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:20.494117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.494102    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d59ee3e-419f-4f34-b4f7-095287a8f640-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:20.890302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.890212    5659 generic.go:356] "Generic (PLEG): container finished" podID="5d59ee3e-419f-4f34-b4f7-095287a8f640" containerID="456ca519eadf85ef305435284d650284f37055ffb8ce30c7d3d84a974c576a33" exitCode=0
Nov 03 20:09:20.890302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.890271    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"
Nov 03 20:09:20.890543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.890297    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd" event={"ID":"5d59ee3e-419f-4f34-b4f7-095287a8f640","Type":"ContainerDied","Data":"456ca519eadf85ef305435284d650284f37055ffb8ce30c7d3d84a974c576a33"}
Nov 03 20:09:20.890543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.890351    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd" event={"ID":"5d59ee3e-419f-4f34-b4f7-095287a8f640","Type":"ContainerDied","Data":"bbd2b28f2360ca4bd286cf4569ffa608756b7d96fce00463f02c33b7b52f3cda"}
Nov 03 20:09:20.890543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.890385    5659 scope.go:122] "RemoveContainer" containerID="456ca519eadf85ef305435284d650284f37055ffb8ce30c7d3d84a974c576a33"
Nov 03 20:09:20.906281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.906261    5659 scope.go:122] "RemoveContainer" containerID="456ca519eadf85ef305435284d650284f37055ffb8ce30c7d3d84a974c576a33"
Nov 03 20:09:20.906626 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:09:20.906604    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456ca519eadf85ef305435284d650284f37055ffb8ce30c7d3d84a974c576a33\": container with ID starting with 456ca519eadf85ef305435284d650284f37055ffb8ce30c7d3d84a974c576a33 not found: ID does not exist" containerID="456ca519eadf85ef305435284d650284f37055ffb8ce30c7d3d84a974c576a33"
Nov 03 20:09:20.906693 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.906635    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456ca519eadf85ef305435284d650284f37055ffb8ce30c7d3d84a974c576a33"} err="failed to get container status \"456ca519eadf85ef305435284d650284f37055ffb8ce30c7d3d84a974c576a33\": rpc error: code = NotFound desc = could not find container \"456ca519eadf85ef305435284d650284f37055ffb8ce30c7d3d84a974c576a33\": container with ID starting with 456ca519eadf85ef305435284d650284f37055ffb8ce30c7d3d84a974c576a33 not found: ID does not exist"
Nov 03 20:09:20.921564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.921541    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"]
Nov 03 20:09:20.928196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.928160    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"]
Nov 03 20:09:20.928425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.928401    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerName="istio-proxy" containerID="cri-o://5fb192bae447d3fa54f9532ef5461e720b84752672a1e0c949d5b1988e2a88fe" gracePeriod=30
Nov 03 20:09:20.932301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:20.932135    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-f154ac/auth-test-model-f154ac-kserve-6d7bf87b85-lwpsd"]
Nov 03 20:09:21.127505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.117693    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"]
Nov 03 20:09:21.127505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.118640    5659 cpu_manager.go:414] "containerMap: removing container" podUID="5d59ee3e-419f-4f34-b4f7-095287a8f640" containerName="main"
Nov 03 20:09:21.127505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.118660    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d59ee3e-419f-4f34-b4f7-095287a8f640" containerName="main"
Nov 03 20:09:21.127505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.118799    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d59ee3e-419f-4f34-b4f7-095287a8f640" containerName="main"
Nov 03 20:09:21.161569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.154798    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:21.198539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.190535    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"]
Nov 03 20:09:21.296676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.288687    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-46a6d7-serving-cert\""
Nov 03 20:09:21.335028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.334996    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:21.353069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.352926    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-metrics-certs\") pod \"maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:21.353069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.353002    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:21.353272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.353211    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821779e9-42f1-4acc-8771-39d4e55f9abf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:21.353399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.353379    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2hpd\" (UniqueName: \"kubernetes.io/projected/821779e9-42f1-4acc-8771-39d4e55f9abf-kube-api-access-g2hpd\") pod \"maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:21.475180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.457778    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-metrics-certs\") pod \"maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:21.475180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.472710    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:21.475180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.472811    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821779e9-42f1-4acc-8771-39d4e55f9abf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:21.475180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.472901    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2hpd\" (UniqueName: \"kubernetes.io/projected/821779e9-42f1-4acc-8771-39d4e55f9abf-kube-api-access-g2hpd\") pod \"maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:21.475180 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:09:21.473323    5659 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-serving-cert: secret "maas-api-e2e-extmod-b-46a6d7-serving-cert" not found
Nov 03 20:09:21.475180 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:09:21.473385    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-maas-api-tls podName:821779e9-42f1-4acc-8771-39d4e55f9abf nodeName:}" failed. No retries permitted until 2026-11-03 20:09:21.973365963 +0000 UTC m=+1914.128268373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-maas-api-tls") pod "maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879" (UID: "821779e9-42f1-4acc-8771-39d4e55f9abf") : secret "maas-api-e2e-extmod-b-46a6d7-serving-cert" not found
Nov 03 20:09:21.475180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.474304    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821779e9-42f1-4acc-8771-39d4e55f9abf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:21.532276 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.531883    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2hpd\" (UniqueName: \"kubernetes.io/projected/821779e9-42f1-4acc-8771-39d4e55f9abf-kube-api-access-g2hpd\") pod \"maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:21.534893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.532881    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-metrics-certs\") pod \"maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:21.553212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.538692    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"]
Nov 03 20:09:21.553212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.539000    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy" containerID="cri-o://db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541" gracePeriod=30
Nov 03 20:09:21.558432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.557425    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:21.604113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.604087    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 20:09:21.676280 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.676245    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-envoy\") pod \"e2b7fd36-49dc-44d2-8b0b-284713245655\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") "
Nov 03 20:09:21.676476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.676353    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-kube-api-access-knsfv\" (UniqueName: \"kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-kube-api-access-knsfv\") pod \"e2b7fd36-49dc-44d2-8b0b-284713245655\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") "
Nov 03 20:09:21.676476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.676400    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-token\" (UniqueName: \"kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-token\") pod \"e2b7fd36-49dc-44d2-8b0b-284713245655\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") "
Nov 03 20:09:21.676476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.676442    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-data\") pod \"e2b7fd36-49dc-44d2-8b0b-284713245655\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") "
Nov 03 20:09:21.676684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.676554    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-socket\") pod \"e2b7fd36-49dc-44d2-8b0b-284713245655\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") "
Nov 03 20:09:21.676684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.676632    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-certs\") pod \"e2b7fd36-49dc-44d2-8b0b-284713245655\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") "
Nov 03 20:09:21.676684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.676672    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-podinfo\") pod \"e2b7fd36-49dc-44d2-8b0b-284713245655\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") "
Nov 03 20:09:21.676844 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.676709    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-credential-socket\") pod \"e2b7fd36-49dc-44d2-8b0b-284713245655\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") "
Nov 03 20:09:21.676915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.676898    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2b7fd36-49dc-44d2-8b0b-284713245655-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b7fd36-49dc-44d2-8b0b-284713245655-istiod-ca-cert\") pod \"e2b7fd36-49dc-44d2-8b0b-284713245655\" (UID: \"e2b7fd36-49dc-44d2-8b0b-284713245655\") "
Nov 03 20:09:21.676970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.676937    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-data" pod "e2b7fd36-49dc-44d2-8b0b-284713245655" (UID: "e2b7fd36-49dc-44d2-8b0b-284713245655"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:21.677570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.677545    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:21.677890 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.677867    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b7fd36-49dc-44d2-8b0b-284713245655-istiod-ca-cert" pod "e2b7fd36-49dc-44d2-8b0b-284713245655" (UID: "e2b7fd36-49dc-44d2-8b0b-284713245655"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:09:21.678124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.678105    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-credential-socket" pod "e2b7fd36-49dc-44d2-8b0b-284713245655" (UID: "e2b7fd36-49dc-44d2-8b0b-284713245655"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:21.678306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.678278    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-socket" pod "e2b7fd36-49dc-44d2-8b0b-284713245655" (UID: "e2b7fd36-49dc-44d2-8b0b-284713245655"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:21.678377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.678358    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-certs" pod "e2b7fd36-49dc-44d2-8b0b-284713245655" (UID: "e2b7fd36-49dc-44d2-8b0b-284713245655"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:21.680268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.680224    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-kube-api-access-knsfv" pod "e2b7fd36-49dc-44d2-8b0b-284713245655" (UID: "e2b7fd36-49dc-44d2-8b0b-284713245655"). InnerVolumeSpecName "kube-api-access-knsfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:21.681024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.681004    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-envoy" pod "e2b7fd36-49dc-44d2-8b0b-284713245655" (UID: "e2b7fd36-49dc-44d2-8b0b-284713245655"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:21.681598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.681571    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-podinfo" pod "e2b7fd36-49dc-44d2-8b0b-284713245655" (UID: "e2b7fd36-49dc-44d2-8b0b-284713245655"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:09:21.681787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.681764    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-token" pod "e2b7fd36-49dc-44d2-8b0b-284713245655" (UID: "e2b7fd36-49dc-44d2-8b0b-284713245655"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:21.698272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.698251    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:21.779064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.778976    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:21.779064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.779011    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b7fd36-49dc-44d2-8b0b-284713245655-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:21.779064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.779024    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:21.779064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.779037    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knsfv\" (UniqueName: \"kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-kube-api-access-knsfv\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:21.779064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.779051    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:21.779064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.779063    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:21.779389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.779076    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b7fd36-49dc-44d2-8b0b-284713245655-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:21.779389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.779087    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b7fd36-49dc-44d2-8b0b-284713245655-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:21.950506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.950087    5659 generic.go:356] "Generic (PLEG): container finished" podID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerID="1e63db2684ed19d8085bb80b751e0fec207dff5efc771072776158159b245b15" exitCode=0
Nov 03 20:09:21.950506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.950222    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" event={"ID":"e2b7fd36-49dc-44d2-8b0b-284713245655","Type":"ContainerDied","Data":"1e63db2684ed19d8085bb80b751e0fec207dff5efc771072776158159b245b15"}
Nov 03 20:09:21.950506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.950273    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78" event={"ID":"e2b7fd36-49dc-44d2-8b0b-284713245655","Type":"ContainerDied","Data":"747127f7d6cf4b9dc4877c7f99c889a52a82992fc3fef6cb56dd51ef5ffa59cb"}
Nov 03 20:09:21.950506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.950304    5659 scope.go:122] "RemoveContainer" containerID="1e63db2684ed19d8085bb80b751e0fec207dff5efc771072776158159b245b15"
Nov 03 20:09:21.950506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.950451    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"
Nov 03 20:09:21.988079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:21.988041    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:22.013149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:22.010323    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d59ee3e-419f-4f34-b4f7-095287a8f640" path="/var/lib/kubelet/pods/5d59ee3e-419f-4f34-b4f7-095287a8f640/volumes"
Nov 03 20:09:22.014898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:22.014865    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:22.039254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:22.039140    5659 scope.go:122] "RemoveContainer" containerID="1e63db2684ed19d8085bb80b751e0fec207dff5efc771072776158159b245b15"
Nov 03 20:09:22.040254 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:09:22.040169    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e63db2684ed19d8085bb80b751e0fec207dff5efc771072776158159b245b15\": container with ID starting with 1e63db2684ed19d8085bb80b751e0fec207dff5efc771072776158159b245b15 not found: ID does not exist" containerID="1e63db2684ed19d8085bb80b751e0fec207dff5efc771072776158159b245b15"
Nov 03 20:09:22.040254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:22.040212    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e63db2684ed19d8085bb80b751e0fec207dff5efc771072776158159b245b15"} err="failed to get container status \"1e63db2684ed19d8085bb80b751e0fec207dff5efc771072776158159b245b15\": rpc error: code = NotFound desc = could not find container \"1e63db2684ed19d8085bb80b751e0fec207dff5efc771072776158159b245b15\": container with ID starting with 1e63db2684ed19d8085bb80b751e0fec207dff5efc771072776158159b245b15 not found: ID does not exist"
Nov 03 20:09:22.096458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:22.096420    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"]
Nov 03 20:09:22.099987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:22.099959    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-gkr78"]
Nov 03 20:09:22.260427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:22.260338    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:22.499658 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:22.499621    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:22.704539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:22.704273    5659 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:09:22.722505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:22.718113    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"]
Nov 03 20:09:22.960228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:22.960197    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879" event={"ID":"821779e9-42f1-4acc-8771-39d4e55f9abf","Type":"ContainerStarted","Data":"028621ac3ce088b039131168d66624b61e7928bf0d7de6efcf5fa0df098525ea"}
Nov 03 20:09:23.327592 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:23.327562    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:23.972538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:23.972504    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" path="/var/lib/kubelet/pods/e2b7fd36-49dc-44d2-8b0b-284713245655/volumes"
Nov 03 20:09:23.972829 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:23.972815    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:23.972903 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:23.972863    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879" event={"ID":"821779e9-42f1-4acc-8771-39d4e55f9abf","Type":"ContainerStarted","Data":"dbea45d6082933388ab4950cf968cfdc64349c709b4c710b4f6ef10bde67e4b0"}
Nov 03 20:09:23.992132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:23.992077    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879" podStartSLOduration=2.572044337 podStartE2EDuration="2.992060739s" totalImagesPullingTime="420.016402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:23.991928694 +0000 UTC m=+1916.146831112" watchObservedRunningTime="2026-11-03 20:09:23.992060739 +0000 UTC m=+1916.146963155"
Nov 03 20:09:24.149503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:24.149456    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:24.870274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:24.870232    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:25.308728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:25.308699    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:25.638597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:25.638514    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:26.215927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.215901    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:09:26.323599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.323559    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-envoy\") pod \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") "
Nov 03 20:09:26.323809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.323613    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-credential-socket\") pod \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") "
Nov 03 20:09:26.323809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.323640    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-kube-api-access-xkcv6\" (UniqueName: \"kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-kube-api-access-xkcv6\") pod \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") "
Nov 03 20:09:26.323809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.323703    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-certs\") pod \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") "
Nov 03 20:09:26.323809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.323745    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-podinfo\") pod \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") "
Nov 03 20:09:26.324035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.323818    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-socket\") pod \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") "
Nov 03 20:09:26.324035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.323902    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-token\" (UniqueName: \"kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-token\") pod \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") "
Nov 03 20:09:26.324035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.323954    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-data\") pod \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") "
Nov 03 20:09:26.324035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.323982    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-credential-socket" pod "09a6df1c-1558-4b8c-b588-59b76dc7ffdd" (UID: "09a6df1c-1558-4b8c-b588-59b76dc7ffdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:26.324253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.324056    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-certs" pod "09a6df1c-1558-4b8c-b588-59b76dc7ffdd" (UID: "09a6df1c-1558-4b8c-b588-59b76dc7ffdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:26.324253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.324121    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istiod-ca-cert\") pod \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\" (UID: \"09a6df1c-1558-4b8c-b588-59b76dc7ffdd\") "
Nov 03 20:09:26.324253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.324169    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-socket" pod "09a6df1c-1558-4b8c-b588-59b76dc7ffdd" (UID: "09a6df1c-1558-4b8c-b588-59b76dc7ffdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:26.324527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.324285    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-data" pod "09a6df1c-1558-4b8c-b588-59b76dc7ffdd" (UID: "09a6df1c-1558-4b8c-b588-59b76dc7ffdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:26.324580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.324559    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istiod-ca-cert" pod "09a6df1c-1558-4b8c-b588-59b76dc7ffdd" (UID: "09a6df1c-1558-4b8c-b588-59b76dc7ffdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:09:26.324785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.324767    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.324842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.324787    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.324842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.324797    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.324842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.324806    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.324842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.324815    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.326520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.326457    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-token" pod "09a6df1c-1558-4b8c-b588-59b76dc7ffdd" (UID: "09a6df1c-1558-4b8c-b588-59b76dc7ffdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:26.326646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.326579    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-envoy" pod "09a6df1c-1558-4b8c-b588-59b76dc7ffdd" (UID: "09a6df1c-1558-4b8c-b588-59b76dc7ffdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:26.326646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.326606    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-kube-api-access-xkcv6" pod "09a6df1c-1558-4b8c-b588-59b76dc7ffdd" (UID: "09a6df1c-1558-4b8c-b588-59b76dc7ffdd"). InnerVolumeSpecName "kube-api-access-xkcv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:26.326729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.326681    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-podinfo" pod "09a6df1c-1558-4b8c-b588-59b76dc7ffdd" (UID: "09a6df1c-1558-4b8c-b588-59b76dc7ffdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:09:26.426700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.426670    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.426700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.426697    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.426700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.426707    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.426952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.426716    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkcv6\" (UniqueName: \"kubernetes.io/projected/09a6df1c-1558-4b8c-b588-59b76dc7ffdd-kube-api-access-xkcv6\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.449826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.449707    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv" podUID="1c26616c-2785-4639-8c41-73b452fe827f" containerName="main" containerID="cri-o://0a2f41786727f2a67b3e4697ab403cf19716b5de4c0c3d876af155de65330969" gracePeriod=45
Nov 03 20:09:26.459956 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.459904    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:26.605045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.603875    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:26.671067 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:09:26.670746    5659 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe201d89_2f4b_47f8_8fa5_983a86e2f87f.slice/crio-db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe201d89_2f4b_47f8_8fa5_983a86e2f87f.slice/crio-conmon-db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:09:26.728809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.728784    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:09:26.830474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.830438    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-tmp-dir\") pod \"1c26616c-2785-4639-8c41-73b452fe827f\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") "
Nov 03 20:09:26.830680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.830576    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c26616c-2785-4639-8c41-73b452fe827f-kube-api-access-fnsvz\" (UniqueName: \"kubernetes.io/projected/1c26616c-2785-4639-8c41-73b452fe827f-kube-api-access-fnsvz\") pod \"1c26616c-2785-4639-8c41-73b452fe827f\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") "
Nov 03 20:09:26.830756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.830700    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-model-cache\") pod \"1c26616c-2785-4639-8c41-73b452fe827f\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") "
Nov 03 20:09:26.830819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.830739    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-tmp-dir" pod "1c26616c-2785-4639-8c41-73b452fe827f" (UID: "1c26616c-2785-4639-8c41-73b452fe827f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:26.831002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.830976    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-model-cache" pod "1c26616c-2785-4639-8c41-73b452fe827f" (UID: "1c26616c-2785-4639-8c41-73b452fe827f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:26.831139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.831089    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-home\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-home\") pod \"1c26616c-2785-4639-8c41-73b452fe827f\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") "
Nov 03 20:09:26.831207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.831192    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c26616c-2785-4639-8c41-73b452fe827f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c26616c-2785-4639-8c41-73b452fe827f-tls-certs\") pod \"1c26616c-2785-4639-8c41-73b452fe827f\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") "
Nov 03 20:09:26.831343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.831320    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-home" pod "1c26616c-2785-4639-8c41-73b452fe827f" (UID: "1c26616c-2785-4639-8c41-73b452fe827f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:26.831418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.831371    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-dshm\") pod \"1c26616c-2785-4639-8c41-73b452fe827f\" (UID: \"1c26616c-2785-4639-8c41-73b452fe827f\") "
Nov 03 20:09:26.831916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.831887    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.831916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.831904    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.831916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.831915    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.833280 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.833257    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c26616c-2785-4639-8c41-73b452fe827f-tls-certs" pod "1c26616c-2785-4639-8c41-73b452fe827f" (UID: "1c26616c-2785-4639-8c41-73b452fe827f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:09:26.833412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.833389    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c26616c-2785-4639-8c41-73b452fe827f-kube-api-access-fnsvz" pod "1c26616c-2785-4639-8c41-73b452fe827f" (UID: "1c26616c-2785-4639-8c41-73b452fe827f"). InnerVolumeSpecName "kube-api-access-fnsvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:26.834149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.834131    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-dshm" pod "1c26616c-2785-4639-8c41-73b452fe827f" (UID: "1c26616c-2785-4639-8c41-73b452fe827f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:26.845819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.845796    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:09:26.933638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.933466    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istiod-ca-cert\") pod \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") "
Nov 03 20:09:26.934299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.933907    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istiod-ca-cert" pod "fe201d89-2f4b-47f8-8fa5-983a86e2f87f" (UID: "fe201d89-2f4b-47f8-8fa5-983a86e2f87f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:09:26.934299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.933947    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-credential-socket\") pod \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") "
Nov 03 20:09:26.934299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.934015    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-envoy\") pod \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") "
Nov 03 20:09:26.934299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.934075    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-token\" (UniqueName: \"kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-token\") pod \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") "
Nov 03 20:09:26.934299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.934112    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-data\") pod \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") "
Nov 03 20:09:26.934299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.934168    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-podinfo\") pod \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") "
Nov 03 20:09:26.934299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.934201    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-certs\") pod \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") "
Nov 03 20:09:26.934299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.934275    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-credential-socket" pod "fe201d89-2f4b-47f8-8fa5-983a86e2f87f" (UID: "fe201d89-2f4b-47f8-8fa5-983a86e2f87f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:26.935687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.934983    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-data" pod "fe201d89-2f4b-47f8-8fa5-983a86e2f87f" (UID: "fe201d89-2f4b-47f8-8fa5-983a86e2f87f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:26.935687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.935032    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-socket\") pod \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") "
Nov 03 20:09:26.935687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.935085    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-kube-api-access-qlzxr\" (UniqueName: \"kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-kube-api-access-qlzxr\") pod \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\" (UID: \"fe201d89-2f4b-47f8-8fa5-983a86e2f87f\") "
Nov 03 20:09:26.935687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.935201    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-certs" pod "fe201d89-2f4b-47f8-8fa5-983a86e2f87f" (UID: "fe201d89-2f4b-47f8-8fa5-983a86e2f87f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:26.935687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.935379    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-socket" pod "fe201d89-2f4b-47f8-8fa5-983a86e2f87f" (UID: "fe201d89-2f4b-47f8-8fa5-983a86e2f87f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:26.936013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.935789    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.936013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.935811    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.936013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.935826    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.936013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.935840    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c26616c-2785-4639-8c41-73b452fe827f-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.936013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.935853    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.936013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.935866    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c26616c-2785-4639-8c41-73b452fe827f-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.936013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.935880    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.936013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.935894    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnsvz\" (UniqueName: \"kubernetes.io/projected/1c26616c-2785-4639-8c41-73b452fe827f-kube-api-access-fnsvz\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:26.940273 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.940251    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-token" pod "fe201d89-2f4b-47f8-8fa5-983a86e2f87f" (UID: "fe201d89-2f4b-47f8-8fa5-983a86e2f87f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:26.940377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.940285    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-kube-api-access-qlzxr" pod "fe201d89-2f4b-47f8-8fa5-983a86e2f87f" (UID: "fe201d89-2f4b-47f8-8fa5-983a86e2f87f"). InnerVolumeSpecName "kube-api-access-qlzxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:26.942437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.942412    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-podinfo" pod "fe201d89-2f4b-47f8-8fa5-983a86e2f87f" (UID: "fe201d89-2f4b-47f8-8fa5-983a86e2f87f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:09:26.943010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:26.942984    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-envoy" pod "fe201d89-2f4b-47f8-8fa5-983a86e2f87f" (UID: "fe201d89-2f4b-47f8-8fa5-983a86e2f87f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:27.003451 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.003370    5659 generic.go:356] "Generic (PLEG): container finished" podID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerID="db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541" exitCode=0
Nov 03 20:09:27.003625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.003609    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"
Nov 03 20:09:27.004305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.004282    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" event={"ID":"fe201d89-2f4b-47f8-8fa5-983a86e2f87f","Type":"ContainerDied","Data":"db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541"}
Nov 03 20:09:27.004377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.004343    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h" event={"ID":"fe201d89-2f4b-47f8-8fa5-983a86e2f87f","Type":"ContainerDied","Data":"35bce39461f58bef9d89ea1e87dd754db1b0c7cc15cd7dec5437086f4e2241a0"}
Nov 03 20:09:27.004430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.004374    5659 scope.go:122] "RemoveContainer" containerID="db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541"
Nov 03 20:09:27.006806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.006784    5659 generic.go:356] "Generic (PLEG): container finished" podID="1c26616c-2785-4639-8c41-73b452fe827f" containerID="0a2f41786727f2a67b3e4697ab403cf19716b5de4c0c3d876af155de65330969" exitCode=0
Nov 03 20:09:27.007060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.006938    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"
Nov 03 20:09:27.007324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.006952    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv" event={"ID":"1c26616c-2785-4639-8c41-73b452fe827f","Type":"ContainerDied","Data":"0a2f41786727f2a67b3e4697ab403cf19716b5de4c0c3d876af155de65330969"}
Nov 03 20:09:27.007324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.007288    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv" event={"ID":"1c26616c-2785-4639-8c41-73b452fe827f","Type":"ContainerDied","Data":"7448835309ec8c6a2843c277fbc019d0c4ced257e9efa7d370bf88297acfc2e0"}
Nov 03 20:09:27.016107 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.008991    5659 generic.go:356] "Generic (PLEG): container finished" podID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerID="5fb192bae447d3fa54f9532ef5461e720b84752672a1e0c949d5b1988e2a88fe" exitCode=0
Nov 03 20:09:27.016107 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.009157    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" event={"ID":"09a6df1c-1558-4b8c-b588-59b76dc7ffdd","Type":"ContainerDied","Data":"5fb192bae447d3fa54f9532ef5461e720b84752672a1e0c949d5b1988e2a88fe"}
Nov 03 20:09:27.016107 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.009199    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n" event={"ID":"09a6df1c-1558-4b8c-b588-59b76dc7ffdd","Type":"ContainerDied","Data":"aa2bc2318b3fca8ba7f76748093f5dbfc13fbc59ab2fa099a822cc8230e5d051"}
Nov 03 20:09:27.016107 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.009270    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"
Nov 03 20:09:27.024854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.024543    5659 scope.go:122] "RemoveContainer" containerID="db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541"
Nov 03 20:09:27.024958 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:09:27.024901    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541\": container with ID starting with db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541 not found: ID does not exist" containerID="db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541"
Nov 03 20:09:27.024958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.024939    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541"} err="failed to get container status \"db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541\": rpc error: code = NotFound desc = could not find container \"db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541\": container with ID starting with db0b021e62bcd750819e04734a5b71ce9a6e99a0ed7ed0f46fef7309cba02541 not found: ID does not exist"
Nov 03 20:09:27.025163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.024962    5659 scope.go:122] "RemoveContainer" containerID="0a2f41786727f2a67b3e4697ab403cf19716b5de4c0c3d876af155de65330969"
Nov 03 20:09:27.037988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.037911    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:27.037988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.037938    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:27.037988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.037954    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:27.037988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.037968    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlzxr\" (UniqueName: \"kubernetes.io/projected/fe201d89-2f4b-47f8-8fa5-983a86e2f87f-kube-api-access-qlzxr\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:27.049892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.048828    5659 scope.go:122] "RemoveContainer" containerID="0a2f41786727f2a67b3e4697ab403cf19716b5de4c0c3d876af155de65330969"
Nov 03 20:09:27.049892 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:09:27.049285    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2f41786727f2a67b3e4697ab403cf19716b5de4c0c3d876af155de65330969\": container with ID starting with 0a2f41786727f2a67b3e4697ab403cf19716b5de4c0c3d876af155de65330969 not found: ID does not exist" containerID="0a2f41786727f2a67b3e4697ab403cf19716b5de4c0c3d876af155de65330969"
Nov 03 20:09:27.049892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.049318    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2f41786727f2a67b3e4697ab403cf19716b5de4c0c3d876af155de65330969"} err="failed to get container status \"0a2f41786727f2a67b3e4697ab403cf19716b5de4c0c3d876af155de65330969\": rpc error: code = NotFound desc = could not find container \"0a2f41786727f2a67b3e4697ab403cf19716b5de4c0c3d876af155de65330969\": container with ID starting with 0a2f41786727f2a67b3e4697ab403cf19716b5de4c0c3d876af155de65330969 not found: ID does not exist"
Nov 03 20:09:27.049892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.049342    5659 scope.go:122] "RemoveContainer" containerID="5fb192bae447d3fa54f9532ef5461e720b84752672a1e0c949d5b1988e2a88fe"
Nov 03 20:09:27.053595 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.052870    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"]
Nov 03 20:09:27.058630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.058600    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-43c674/auth-test-model-43c674-kserve-9477cf84f-pf8gv"]
Nov 03 20:09:27.063005 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.062971    5659 scope.go:122] "RemoveContainer" containerID="5fb192bae447d3fa54f9532ef5461e720b84752672a1e0c949d5b1988e2a88fe"
Nov 03 20:09:27.063454 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:09:27.063432    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb192bae447d3fa54f9532ef5461e720b84752672a1e0c949d5b1988e2a88fe\": container with ID starting with 5fb192bae447d3fa54f9532ef5461e720b84752672a1e0c949d5b1988e2a88fe not found: ID does not exist" containerID="5fb192bae447d3fa54f9532ef5461e720b84752672a1e0c949d5b1988e2a88fe"
Nov 03 20:09:27.063571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.063464    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb192bae447d3fa54f9532ef5461e720b84752672a1e0c949d5b1988e2a88fe"} err="failed to get container status \"5fb192bae447d3fa54f9532ef5461e720b84752672a1e0c949d5b1988e2a88fe\": rpc error: code = NotFound desc = could not find container \"5fb192bae447d3fa54f9532ef5461e720b84752672a1e0c949d5b1988e2a88fe\": container with ID starting with 5fb192bae447d3fa54f9532ef5461e720b84752672a1e0c949d5b1988e2a88fe not found: ID does not exist"
Nov 03 20:09:27.071359 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.071332    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"]
Nov 03 20:09:27.078262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.078168    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:27.079331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.079304    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-w5x6h"]
Nov 03 20:09:27.090491 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.090452    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"]
Nov 03 20:09:27.097708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.097684    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rvv4n"]
Nov 03 20:09:27.227411 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.227376    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:27.971050 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.971019    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" path="/var/lib/kubelet/pods/09a6df1c-1558-4b8c-b588-59b76dc7ffdd/volumes"
Nov 03 20:09:27.971608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.971585    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c26616c-2785-4639-8c41-73b452fe827f" path="/var/lib/kubelet/pods/1c26616c-2785-4639-8c41-73b452fe827f/volumes"
Nov 03 20:09:27.972044 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.972026    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" path="/var/lib/kubelet/pods/fe201d89-2f4b-47f8-8fa5-983a86e2f87f/volumes"
Nov 03 20:09:27.998260 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:27.998220    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:28.923259 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:28.923225    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:29.556505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.552524    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"]
Nov 03 20:09:29.556505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.553567    5659 cpu_manager.go:414] "containerMap: removing container" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy"
Nov 03 20:09:29.556505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.553588    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy"
Nov 03 20:09:29.556505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.553650    5659 cpu_manager.go:414] "containerMap: removing container" podUID="1c26616c-2785-4639-8c41-73b452fe827f" containerName="main"
Nov 03 20:09:29.556505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.553659    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c26616c-2785-4639-8c41-73b452fe827f" containerName="main"
Nov 03 20:09:29.556505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.553683    5659 cpu_manager.go:414] "containerMap: removing container" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerName="istio-proxy"
Nov 03 20:09:29.556505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.553691    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerName="istio-proxy"
Nov 03 20:09:29.556505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.553721    5659 cpu_manager.go:414] "containerMap: removing container" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy"
Nov 03 20:09:29.556505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.553730    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy"
Nov 03 20:09:29.556505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.553884    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c26616c-2785-4639-8c41-73b452fe827f" containerName="main"
Nov 03 20:09:29.556505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.553898    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a6df1c-1558-4b8c-b588-59b76dc7ffdd" containerName="istio-proxy"
Nov 03 20:09:29.556505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.553911    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b7fd36-49dc-44d2-8b0b-284713245655" containerName="istio-proxy"
Nov 03 20:09:29.556505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.553921    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe201d89-2f4b-47f8-8fa5-983a86e2f87f" containerName="istio-proxy"
Nov 03 20:09:29.592513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.564750    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.592513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.581733    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"]
Nov 03 20:09:29.599512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.594849    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bcfde8-openshift-default-dockercfg-6xbwb\""
Nov 03 20:09:29.739653 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.739617    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-istio-token\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.739837 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.739669    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee89175-06af-4e58-8b04-bd892b530f1e-istio-podinfo\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.739837 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.739756    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-certs\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.739919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.739865    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-envoy\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.739951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.739922    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bsl\" (UniqueName: \"kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-kube-api-access-94bsl\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.739988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.739951    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-data\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.740037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.740022    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-credential-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.740082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.740071    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.740117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.740105    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee89175-06af-4e58-8b04-bd892b530f1e-istiod-ca-cert\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.840976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.840936    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-credential-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.841094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.841033    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.841094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.841087    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee89175-06af-4e58-8b04-bd892b530f1e-istiod-ca-cert\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.841225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.841171    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-istio-token\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.841281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.841237    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee89175-06af-4e58-8b04-bd892b530f1e-istio-podinfo\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.841335 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.841283    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-certs\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.841393 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.841348    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-credential-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.841449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.841400    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.841449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.841405    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-envoy\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.841601 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.841553    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bsl\" (UniqueName: \"kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-kube-api-access-94bsl\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.841652 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.841602    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-data\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.842011 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.841983    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-certs\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.842132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.842108    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee89175-06af-4e58-8b04-bd892b530f1e-istiod-ca-cert\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.842198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.842183    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-data\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.844107 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.844079    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-envoy\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.844454 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.844432    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee89175-06af-4e58-8b04-bd892b530f1e-istio-podinfo\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.888533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.888497    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-istio-token\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:29.889404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:29.889381    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bsl\" (UniqueName: \"kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-kube-api-access-94bsl\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:30.012504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:30.006770    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:30.024505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:30.018835    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:09:30.490797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:30.490745    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:30.526548 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:09:30.526471    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee89175_06af_4e58_8b04_bd892b530f1e.slice/crio-60d196ac4d6ac26a2b01fdbd9a19273819d066c6fec7abc8294bf8c12fcd5892 WatchSource:0}: Error finding container 60d196ac4d6ac26a2b01fdbd9a19273819d066c6fec7abc8294bf8c12fcd5892: Status 404 returned error can't find the container with id 60d196ac4d6ac26a2b01fdbd9a19273819d066c6fec7abc8294bf8c12fcd5892
Nov 03 20:09:30.531337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:30.531264    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:30.531464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:30.531431    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:30.531643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:30.531588    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:30.532645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:30.532601    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"]
Nov 03 20:09:30.894128 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:30.894047    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"
Nov 03 20:09:30.968503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:30.967145    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:31.099369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:31.099335    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf" event={"ID":"bee89175-06af-4e58-8b04-bd892b530f1e","Type":"ContainerStarted","Data":"8b95a5948ef9b60fc35d20919ab0ad9f743cab22f5c1f57f89e65c64554c74e7"}
Nov 03 20:09:31.099851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:31.099397    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf" event={"ID":"bee89175-06af-4e58-8b04-bd892b530f1e","Type":"ContainerStarted","Data":"60d196ac4d6ac26a2b01fdbd9a19273819d066c6fec7abc8294bf8c12fcd5892"}
Nov 03 20:09:31.114682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:31.114649    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:31.128319 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:31.128252    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf" podStartSLOduration=2.128235029 podStartE2EDuration="2.128235029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:31.124397718 +0000 UTC m=+1923.279300136" watchObservedRunningTime="2026-11-03 20:09:31.128235029 +0000 UTC m=+1923.283137449"
Nov 03 20:09:31.590156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:31.590115    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:32.007494 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.007442    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:32.008826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.008799    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:09:32.008963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.008845    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf" podUID="bee89175-06af-4e58-8b04-bd892b530f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 20:09:32.108530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.103038    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:32.266439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.266337    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:32.273969 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.272881    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"]
Nov 03 20:09:32.285562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.284882    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"
Nov 03 20:09:32.300964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.297139    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0e1546\""
Nov 03 20:09:32.304212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.304180    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"]
Nov 03 20:09:32.566003 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.538475    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:32.569513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.567028    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11bf2ea4-c852-4165-854c-ce2ad2018274-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt\" (UID: \"11bf2ea4-c852-4165-854c-ce2ad2018274\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"
Nov 03 20:09:32.569513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.567272    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjhww\" (UniqueName: \"kubernetes.io/projected/11bf2ea4-c852-4165-854c-ce2ad2018274-kube-api-access-bjhww\") pod \"payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt\" (UID: \"11bf2ea4-c852-4165-854c-ce2ad2018274\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"
Nov 03 20:09:32.673951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.673907    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjhww\" (UniqueName: \"kubernetes.io/projected/11bf2ea4-c852-4165-854c-ce2ad2018274-kube-api-access-bjhww\") pod \"payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt\" (UID: \"11bf2ea4-c852-4165-854c-ce2ad2018274\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"
Nov 03 20:09:32.674828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.674798    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11bf2ea4-c852-4165-854c-ce2ad2018274-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt\" (UID: \"11bf2ea4-c852-4165-854c-ce2ad2018274\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"
Nov 03 20:09:32.676174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.676151    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11bf2ea4-c852-4165-854c-ce2ad2018274-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt\" (UID: \"11bf2ea4-c852-4165-854c-ce2ad2018274\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"
Nov 03 20:09:32.728318 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.728169    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjhww\" (UniqueName: \"kubernetes.io/projected/11bf2ea4-c852-4165-854c-ce2ad2018274-kube-api-access-bjhww\") pod \"payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt\" (UID: \"11bf2ea4-c852-4165-854c-ce2ad2018274\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"
Nov 03 20:09:32.867558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.854058    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:32.926528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:32.926079    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"
Nov 03 20:09:33.008421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:33.008384    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:09:33.008605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:33.008441    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf" podUID="bee89175-06af-4e58-8b04-bd892b530f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 20:09:33.275100 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:09:33.275070    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bf2ea4_c852_4165_854c_ce2ad2018274.slice/crio-25d3b4f69bd2a09ea1c07e4f8489e59742277c69f3af69bb47874c855f244464 WatchSource:0}: Error finding container 25d3b4f69bd2a09ea1c07e4f8489e59742277c69f3af69bb47874c855f244464: Status 404 returned error can't find the container with id 25d3b4f69bd2a09ea1c07e4f8489e59742277c69f3af69bb47874c855f244464
Nov 03 20:09:33.277053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:33.277029    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"]
Nov 03 20:09:33.798787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:33.798751    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:34.007623 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.007586    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:09:34.007834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.007643    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf" podUID="bee89175-06af-4e58-8b04-bd892b530f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 20:09:34.179748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.179647    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt" event={"ID":"11bf2ea4-c852-4165-854c-ce2ad2018274","Type":"ContainerStarted","Data":"bb3bccd01f7256713dfd8302ce1f99976de8c6976b1ef3a5497823db6b518b52"}
Nov 03 20:09:34.179748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.179702    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt" event={"ID":"11bf2ea4-c852-4165-854c-ce2ad2018274","Type":"ContainerStarted","Data":"25d3b4f69bd2a09ea1c07e4f8489e59742277c69f3af69bb47874c855f244464"}
Nov 03 20:09:34.179748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.179724    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"
Nov 03 20:09:34.207091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.207032    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt" podStartSLOduration=2.207014096 podStartE2EDuration="2.207014096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:34.200954177 +0000 UTC m=+1926.355856598" watchObservedRunningTime="2026-11-03 20:09:34.207014096 +0000 UTC m=+1926.361916516"
Nov 03 20:09:34.513449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.513417    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"]
Nov 03 20:09:34.517774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.517749    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:34.529258 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.526699    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-98529d-serving-cert\""
Nov 03 20:09:34.559662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.559548    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"]
Nov 03 20:09:34.598356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.598313    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:34.606997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.606968    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e7029d-ce9c-452a-8d22-2b7b12212c7c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:34.607161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.607018    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vvgq\" (UniqueName: \"kubernetes.io/projected/80e7029d-ce9c-452a-8d22-2b7b12212c7c-kube-api-access-5vvgq\") pod \"maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:34.607246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.607172    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:34.607325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.607258    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-metrics-certs\") pod \"maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:34.711277 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.711230    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e7029d-ce9c-452a-8d22-2b7b12212c7c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:34.711457 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.711325    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vvgq\" (UniqueName: \"kubernetes.io/projected/80e7029d-ce9c-452a-8d22-2b7b12212c7c-kube-api-access-5vvgq\") pod \"maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:34.711457 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.711411    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:34.711599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.711498    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-metrics-certs\") pod \"maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:34.711699 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:09:34.711674    5659 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-serving-cert: secret "maas-api-e2e-extmod-a-98529d-serving-cert" not found
Nov 03 20:09:34.711764 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:09:34.711759    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-maas-api-tls podName:80e7029d-ce9c-452a-8d22-2b7b12212c7c nodeName:}" failed. No retries permitted until 2026-11-03 20:09:35.211735353 +0000 UTC m=+1927.366637765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-maas-api-tls") pod "maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5" (UID: "80e7029d-ce9c-452a-8d22-2b7b12212c7c") : secret "maas-api-e2e-extmod-a-98529d-serving-cert" not found
Nov 03 20:09:34.712326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.712303    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e7029d-ce9c-452a-8d22-2b7b12212c7c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:34.714755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.714732    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-metrics-certs\") pod \"maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:34.724973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.724949    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vvgq\" (UniqueName: \"kubernetes.io/projected/80e7029d-ce9c-452a-8d22-2b7b12212c7c-kube-api-access-5vvgq\") pod \"maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:34.763783 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.761152    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:34.889553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:34.889519    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:35.007723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:35.007677    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:09:35.007901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:35.007747    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf" podUID="bee89175-06af-4e58-8b04-bd892b530f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 20:09:35.228604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:35.226560    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:35.255507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:35.250197    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:35.472145 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:35.471939    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:35.555507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:35.543793    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:35.716505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:35.695088    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:35.883037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:35.875239    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"]
Nov 03 20:09:36.007745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:36.007711    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:09:36.007931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:36.007767    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf" podUID="bee89175-06af-4e58-8b04-bd892b530f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 20:09:36.234505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:36.234094    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5" event={"ID":"80e7029d-ce9c-452a-8d22-2b7b12212c7c","Type":"ContainerStarted","Data":"1bc6b310615d06852c30fcd528b518aa36428b9b7b198ac619d766fedbedd9bc"}
Nov 03 20:09:36.470298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:36.470274    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:37.007940 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:37.007905    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:09:37.008330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:37.007959    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf" podUID="bee89175-06af-4e58-8b04-bd892b530f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 20:09:37.092055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:37.092015    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:37.225672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:37.225636    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:37.244414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:37.244375    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5" event={"ID":"80e7029d-ce9c-452a-8d22-2b7b12212c7c","Type":"ContainerStarted","Data":"31d4610d3e4a3d53c5fd2dbecec267841d9fa3e75680920c95140138a79e68de"}
Nov 03 20:09:37.244657 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:37.244626    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:37.266035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:37.265930    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5" podStartSLOduration=2.767480772 podStartE2EDuration="3.265916652s" totalImagesPullingTime="498.43588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:37.264266442 +0000 UTC m=+1929.419168873" watchObservedRunningTime="2026-11-03 20:09:37.265916652 +0000 UTC m=+1929.420819071"
Nov 03 20:09:37.355919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:37.355886    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:38.012779 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:38.012752    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:38.013156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:38.013034    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:38.013733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:38.013710    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:09:38.052007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:38.051976    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:38.735993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:38.735963    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:38.912745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:38.912716    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:39.405191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:39.405159    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:40.064579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:40.064546    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:40.845506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:40.842180    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:41.731326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:41.731281    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:42.536507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:42.527043    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:43.256408 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:43.256378    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:09:43.356184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:43.356154    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:43.511015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:43.510937    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:44.043403 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:44.043364    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:44.673352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:44.673320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:44.809435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:44.809399    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:44.957590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:44.957564    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:45.100387 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:45.100360    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:45.204079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:45.204051    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"
Nov 03 20:09:45.217845 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:09:45.217762    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:09:45.279107 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:45.278891    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:45.546560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:45.533917    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:45.701850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:45.701823    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:45.983005 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:45.982968    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:46.156023 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:46.155979    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:46.452684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:46.452600    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:46.585983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:46.585709    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:46.780710 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:46.780676    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:47.050577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:47.049755    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:47.815306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:47.815251    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:48.625289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:48.625260    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:49.278801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.278765    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"]
Nov 03 20:09:49.284016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.283990    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:49.292579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.291001    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0e1546-serving-cert\""
Nov 03 20:09:49.295889 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.295860    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"]
Nov 03 20:09:49.306269 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.306232    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:49.306393 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.306350    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:49.306441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.306410    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-metrics-certs\") pod \"maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:49.306596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.306573    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcgfw\" (UniqueName: \"kubernetes.io/projected/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-kube-api-access-dcgfw\") pod \"maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:49.345537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.345176    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:49.408477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.408440    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:49.408703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.408578    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:49.408703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.408634    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-metrics-certs\") pod \"maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:49.408824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.408784    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcgfw\" (UniqueName: \"kubernetes.io/projected/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-kube-api-access-dcgfw\") pod \"maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:49.409254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.409228    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:49.411191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.411171    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:49.411286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.411195    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-metrics-certs\") pod \"maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:49.421807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.421774    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcgfw\" (UniqueName: \"kubernetes.io/projected/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-kube-api-access-dcgfw\") pod \"maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:49.485115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.485077    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:49.599551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:49.599443    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:50.001851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:50.001822    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"]
Nov 03 20:09:50.006585 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:09:50.006557    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7349216a_564a_4fea_8cd8_4a8e8a6f3b2c.slice/crio-524d4db1b759a071ead39ddbebef3ef8d0ac891ef075db516e136de3e5ce50b0 WatchSource:0}: Error finding container 524d4db1b759a071ead39ddbebef3ef8d0ac891ef075db516e136de3e5ce50b0: Status 404 returned error can't find the container with id 524d4db1b759a071ead39ddbebef3ef8d0ac891ef075db516e136de3e5ce50b0
Nov 03 20:09:50.150577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:50.150540    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:50.374171 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:50.374085    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg" event={"ID":"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c","Type":"ContainerStarted","Data":"524d4db1b759a071ead39ddbebef3ef8d0ac891ef075db516e136de3e5ce50b0"}
Nov 03 20:09:51.071941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:51.071906    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:51.476505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:51.464304    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg" event={"ID":"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c","Type":"ContainerStarted","Data":"bab3ec68828d0de8a5a4fe773ce50cc40250bc44bddd469c9be3e892230825f6"}
Nov 03 20:09:51.477267 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:51.477240    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:51.583504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:51.572362    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg" podStartSLOduration=2.002830523 podStartE2EDuration="2.572347615s" totalImagesPullingTime="569.517092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:51.552316814 +0000 UTC m=+1943.707219333" watchObservedRunningTime="2026-11-03 20:09:51.572347615 +0000 UTC m=+1943.727250031"
Nov 03 20:09:52.023983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:52.023952    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:52.830718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:52.830683    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:53.414053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:53.414015    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:53.555992 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:53.555956    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:53.922517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:53.907421    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:54.795111 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:54.795072    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:55.410146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:55.410065    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:55.786300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:55.785904    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:56.626305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:56.626256    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:56.930900 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:56.930825    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:57.419106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:57.419071    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:57.479542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:57.479514    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:09:58.144025 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:58.143993    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:58.722851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:58.722788    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:58.919511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:58.919437    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:59.943981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:09:59.943944    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:00.274063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:00.274028    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:00.416416 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:00.416373    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:01.154248 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.154210    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:01.557900 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.557866    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:01.589677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.589643    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"]
Nov 03 20:10:01.595009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.594983    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"
Nov 03 20:10:01.607384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.605817    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0e4ce7\""
Nov 03 20:10:01.607384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.606469    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0e4ce7-dockercfg-2sn5h\""
Nov 03 20:10:01.613391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.611719    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"]
Nov 03 20:10:01.642420 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.642380    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"]
Nov 03 20:10:01.648691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.648658    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"
Nov 03 20:10:01.654562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.654499    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"]
Nov 03 20:10:01.693588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.693551    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:01.714378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.714129    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c397c76f-9780-496f-bb87-372a06c94d40-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng\" (UID: \"c397c76f-9780-496f-bb87-372a06c94d40\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"
Nov 03 20:10:01.746009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.745973    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn8tw\" (UniqueName: \"kubernetes.io/projected/c397c76f-9780-496f-bb87-372a06c94d40-kube-api-access-xn8tw\") pod \"payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng\" (UID: \"c397c76f-9780-496f-bb87-372a06c94d40\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"
Nov 03 20:10:01.746238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.746215    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b039d7a8-14c3-4195-9967-121992120988-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq\" (UID: \"b039d7a8-14c3-4195-9967-121992120988\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"
Nov 03 20:10:01.759246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.746582    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk5wk\" (UniqueName: \"kubernetes.io/projected/b039d7a8-14c3-4195-9967-121992120988-kube-api-access-lk5wk\") pod \"payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq\" (UID: \"b039d7a8-14c3-4195-9967-121992120988\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"
Nov 03 20:10:01.870895 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.870171    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c397c76f-9780-496f-bb87-372a06c94d40-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng\" (UID: \"c397c76f-9780-496f-bb87-372a06c94d40\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"
Nov 03 20:10:01.870895 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.870349    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn8tw\" (UniqueName: \"kubernetes.io/projected/c397c76f-9780-496f-bb87-372a06c94d40-kube-api-access-xn8tw\") pod \"payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng\" (UID: \"c397c76f-9780-496f-bb87-372a06c94d40\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"
Nov 03 20:10:01.870895 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.870389    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b039d7a8-14c3-4195-9967-121992120988-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq\" (UID: \"b039d7a8-14c3-4195-9967-121992120988\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"
Nov 03 20:10:01.870895 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.870617    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk5wk\" (UniqueName: \"kubernetes.io/projected/b039d7a8-14c3-4195-9967-121992120988-kube-api-access-lk5wk\") pod \"payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq\" (UID: \"b039d7a8-14c3-4195-9967-121992120988\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"
Nov 03 20:10:01.903504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.871936    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c397c76f-9780-496f-bb87-372a06c94d40-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng\" (UID: \"c397c76f-9780-496f-bb87-372a06c94d40\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"
Nov 03 20:10:01.903504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.873010    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b039d7a8-14c3-4195-9967-121992120988-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq\" (UID: \"b039d7a8-14c3-4195-9967-121992120988\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"
Nov 03 20:10:01.942518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.942422    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn8tw\" (UniqueName: \"kubernetes.io/projected/c397c76f-9780-496f-bb87-372a06c94d40-kube-api-access-xn8tw\") pod \"payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng\" (UID: \"c397c76f-9780-496f-bb87-372a06c94d40\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"
Nov 03 20:10:01.962531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.957379    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk5wk\" (UniqueName: \"kubernetes.io/projected/b039d7a8-14c3-4195-9967-121992120988-kube-api-access-lk5wk\") pod \"payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq\" (UID: \"b039d7a8-14c3-4195-9967-121992120988\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"
Nov 03 20:10:01.981336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:01.981292    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"
Nov 03 20:10:02.219211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:02.219177    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"
Nov 03 20:10:02.263877 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:02.263845    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:02.293519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:02.292841    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"]
Nov 03 20:10:02.311131 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:10:02.309849    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb039d7a8_14c3_4195_9967_121992120988.slice/crio-137c90648dc4d2b28a9d389c7921e2a78ec2dbf1643312a90f4af312b4ec15d5 WatchSource:0}: Error finding container 137c90648dc4d2b28a9d389c7921e2a78ec2dbf1643312a90f4af312b4ec15d5: Status 404 returned error can't find the container with id 137c90648dc4d2b28a9d389c7921e2a78ec2dbf1643312a90f4af312b4ec15d5
Nov 03 20:10:02.543569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:02.543531    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"]
Nov 03 20:10:02.544006 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:10:02.543978    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc397c76f_9780_496f_bb87_372a06c94d40.slice/crio-6304dd6feecf6bd923f4aed74075431e71df5cc6956991faea28296229d8d15e WatchSource:0}: Error finding container 6304dd6feecf6bd923f4aed74075431e71df5cc6956991faea28296229d8d15e: Status 404 returned error can't find the container with id 6304dd6feecf6bd923f4aed74075431e71df5cc6956991faea28296229d8d15e
Nov 03 20:10:02.626364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:02.626326    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq" event={"ID":"b039d7a8-14c3-4195-9967-121992120988","Type":"ContainerStarted","Data":"54f85a05be55d25d64456cc4eac02566b1220c8a13a63d3b9595ed301f382bff"}
Nov 03 20:10:02.626540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:02.626393    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq" event={"ID":"b039d7a8-14c3-4195-9967-121992120988","Type":"ContainerStarted","Data":"137c90648dc4d2b28a9d389c7921e2a78ec2dbf1643312a90f4af312b4ec15d5"}
Nov 03 20:10:02.626540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:02.626441    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"
Nov 03 20:10:02.628532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:02.628472    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng" event={"ID":"c397c76f-9780-496f-bb87-372a06c94d40","Type":"ContainerStarted","Data":"6304dd6feecf6bd923f4aed74075431e71df5cc6956991faea28296229d8d15e"}
Nov 03 20:10:02.653522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:02.653439    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq" podStartSLOduration=1.653417076 podStartE2EDuration="1.653417076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:02.65044354 +0000 UTC m=+1954.805345960" watchObservedRunningTime="2026-11-03 20:10:02.653417076 +0000 UTC m=+1954.808319520"
Nov 03 20:10:03.127028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:03.126997    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:03.638829 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:03.638790    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng" event={"ID":"c397c76f-9780-496f-bb87-372a06c94d40","Type":"ContainerStarted","Data":"c2a7c76a41b1dbb4ec664637ef28c68f36b2e5e7f1f16102d83ab00b6c7a606e"}
Nov 03 20:10:03.639277 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:03.638860    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"
Nov 03 20:10:03.658274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:03.658207    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng" podStartSLOduration=2.658185703 podStartE2EDuration="2.658185703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:03.656897748 +0000 UTC m=+1955.811800159" watchObservedRunningTime="2026-11-03 20:10:03.658185703 +0000 UTC m=+1955.813088123"
Nov 03 20:10:04.080509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:04.072997    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:04.855685 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:04.855642    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:05.576448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:05.576393    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:06.453149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:06.453108    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:06.677654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:06.677621    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:07.445623 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:07.445589    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:08.005427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:08.005383    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:08.199416 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:08.199383    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:08.521561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:08.516850    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:08.845254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:08.845174    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:09.162134 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:09.162044    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:09.938004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:09.937974    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:10.756567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:10.756528    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:11.525651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:11.525552    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:11.748171 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:11.748139    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:12.662822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.662790    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:12.735524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.735468    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"]
Nov 03 20:10:12.741903 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.741879    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.755902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.754875    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-36556d-openshift-default-dockercfg-hnpg9\""
Nov 03 20:10:12.763978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.763947    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"]
Nov 03 20:10:12.847438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.847395    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.847646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.847551    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.847697 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.847663    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.847734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.847707    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.847901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.847884    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a68390b-ddc0-4718-856b-495251cf9647-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.847949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.847919    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ffv\" (UniqueName: \"kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-kube-api-access-p9ffv\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.848070 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.848053    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.848178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.848095    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.848178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.848119    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a68390b-ddc0-4718-856b-495251cf9647-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.950166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.950062    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.950166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.950159    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.950402 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.950262    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.950402 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.950290    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.950526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.950443    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a68390b-ddc0-4718-856b-495251cf9647-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.950740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.950473    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ffv\" (UniqueName: \"kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-kube-api-access-p9ffv\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.950836 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.950816    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.951157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.951129    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.951226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.951187    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.951226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.951211    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.951380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.951333    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.951532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.951511    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.951625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.951243    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a68390b-ddc0-4718-856b-495251cf9647-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.952219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.952157    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a68390b-ddc0-4718-856b-495251cf9647-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.953260 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.953221    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.953723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.953700    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a68390b-ddc0-4718-856b-495251cf9647-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.962788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.962765    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:12.963037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:12.963003    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ffv\" (UniqueName: \"kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-kube-api-access-p9ffv\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-v6sm7\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:13.065496 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.065444    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:13.331199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.331166    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:13.385304 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:10:13.380040    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a68390b_ddc0_4718_856b_495251cf9647.slice/crio-df3fe513cefb4c53109a33f763495baf3851f698600a79e20167a630d4fb7258 WatchSource:0}: Error finding container df3fe513cefb4c53109a33f763495baf3851f698600a79e20167a630d4fb7258: Status 404 returned error can't find the container with id df3fe513cefb4c53109a33f763495baf3851f698600a79e20167a630d4fb7258
Nov 03 20:10:13.388127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.388093    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"]
Nov 03 20:10:13.401883 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.401429    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:13.401883 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.401555    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:13.401883 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.401601    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:13.566579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.566410    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"]
Nov 03 20:10:13.578517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.578163    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.581578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.580992    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"]
Nov 03 20:10:13.642733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.642702    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"
Nov 03 20:10:13.648443 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.648404    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-46a6d7-openshift-default-dockercfg-5jblx\""
Nov 03 20:10:13.691411 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.691370    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.691956 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.691504    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.692440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.692406    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.694016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.693986    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.694175 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.694057    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.694175 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.694083    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb59k\" (UniqueName: \"kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-kube-api-access-cb59k\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.694175 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.694114    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.694175 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.694158    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.694422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.694201    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.795735 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.795700    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.795947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.795821    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.795947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.795850    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.795947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.795876    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb59k\" (UniqueName: \"kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-kube-api-access-cb59k\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.796108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.796016    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.796108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.796081    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.796217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.796128    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.796217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.796181    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.796217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.796183    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.796400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.796266    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.796538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.796517    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.796538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.796529    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.796689 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.796552    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.797057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.797034    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.798529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.798505    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.800009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.799975    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.810132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.810107    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.810529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.810509    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb59k\" (UniqueName: \"kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-kube-api-access-cb59k\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:13.826135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.826101    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7" event={"ID":"9a68390b-ddc0-4718-856b-495251cf9647","Type":"ContainerStarted","Data":"01c0d077cc105a45b32b8af691c352a81b13b88d9ef946868917b9ebb630220e"}
Nov 03 20:10:13.826248 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.826169    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7" event={"ID":"9a68390b-ddc0-4718-856b-495251cf9647","Type":"ContainerStarted","Data":"df3fe513cefb4c53109a33f763495baf3851f698600a79e20167a630d4fb7258"}
Nov 03 20:10:13.857072 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.856969    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7" podStartSLOduration=1.856954158 podStartE2EDuration="1.856954158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:13.853951537 +0000 UTC m=+1966.008853958" watchObservedRunningTime="2026-11-03 20:10:13.856954158 +0000 UTC m=+1966.011856577"
Nov 03 20:10:13.925126 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:13.925083    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:14.074547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.067223    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:14.074547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.067379    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:10:14.074547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.067423    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7" podUID="9a68390b-ddc0-4718-856b-495251cf9647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 20:10:14.237508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.231799    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"]
Nov 03 20:10:14.239378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.238035    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:14.239378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.238137    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:14.239378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.238178    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:14.399367 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.399218    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:14.545637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.537667    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:14.676726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.676693    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"
Nov 03 20:10:14.896791 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.896700    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" event={"ID":"72c08e50-44dd-4bf9-ae14-f24e2672e4c5","Type":"ContainerStarted","Data":"644f3389cf12ca2ff17fb4d8d4e18752e106e72b6619d1aa5fed451577ea0494"}
Nov 03 20:10:14.897498 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.897442    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" event={"ID":"72c08e50-44dd-4bf9-ae14-f24e2672e4c5","Type":"ContainerStarted","Data":"6f806608af35668d035c9e7519f13094bfcb3d5ac48c59b95527c5cebc162652"}
Nov 03 20:10:14.898978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.866453    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:14.927278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.927248    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:14.975577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:14.975514    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" podStartSLOduration=1.975471489 podStartE2EDuration="1.975471489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:14.930828343 +0000 UTC m=+1967.085730763" watchObservedRunningTime="2026-11-03 20:10:14.975471489 +0000 UTC m=+1967.130373911"
Nov 03 20:10:15.057050 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:15.057009    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:15.069505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:15.066854    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:10:15.069505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:15.066912    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7" podUID="9a68390b-ddc0-4718-856b-495251cf9647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 20:10:15.221250 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:10:15.221196    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:10:15.491560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:15.486036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:15.926907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:15.926824    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:10:15.926907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:15.926874    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" podUID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:10:16.069564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.068892    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:10:16.069564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.068956    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7" podUID="9a68390b-ddc0-4718-856b-495251cf9647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 20:10:16.221719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.218656    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:16.324639 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.323963    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"]
Nov 03 20:10:16.329962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.329932    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"]
Nov 03 20:10:16.330256 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.330030    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"
Nov 03 20:10:16.394565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.388798    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b503a8\""
Nov 03 20:10:16.451000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.450960    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-b503a8-dockercfg-d2m4z\""
Nov 03 20:10:16.477395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.470954    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst\" (UID: \"a351e8e7-b09d-44fd-ba3e-8c811db79f8a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"
Nov 03 20:10:16.477395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.471136    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9df\" (UniqueName: \"kubernetes.io/projected/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-kube-api-access-lm9df\") pod \"payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst\" (UID: \"a351e8e7-b09d-44fd-ba3e-8c811db79f8a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"
Nov 03 20:10:16.477395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.472330    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:16.623409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.623361    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst\" (UID: \"a351e8e7-b09d-44fd-ba3e-8c811db79f8a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"
Nov 03 20:10:16.691640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.691608    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9df\" (UniqueName: \"kubernetes.io/projected/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-kube-api-access-lm9df\") pod \"payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst\" (UID: \"a351e8e7-b09d-44fd-ba3e-8c811db79f8a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"
Nov 03 20:10:16.707254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.690972    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst\" (UID: \"a351e8e7-b09d-44fd-ba3e-8c811db79f8a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"
Nov 03 20:10:16.707548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.666571    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:16.752605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.735431    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9df\" (UniqueName: \"kubernetes.io/projected/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-kube-api-access-lm9df\") pod \"payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst\" (UID: \"a351e8e7-b09d-44fd-ba3e-8c811db79f8a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"
Nov 03 20:10:16.925974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.925937    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:10:16.926145 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:16.926012    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" podUID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:10:17.024078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:17.023996    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"
Nov 03 20:10:17.075776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:17.065918    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:10:17.075776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:17.065990    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7" podUID="9a68390b-ddc0-4718-856b-495251cf9647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 20:10:17.469645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:17.469601    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:17.564852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:17.564816    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"]
Nov 03 20:10:17.925547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:17.925418    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:10:17.925547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:17.925467    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" podUID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:10:17.926812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:17.926728    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst" event={"ID":"a351e8e7-b09d-44fd-ba3e-8c811db79f8a","Type":"ContainerStarted","Data":"5622f72b2bcef470edb4281fefb6316e09ca2c2931c2390b5632b5e452628d32"}
Nov 03 20:10:17.926812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:17.926783    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst" event={"ID":"a351e8e7-b09d-44fd-ba3e-8c811db79f8a","Type":"ContainerStarted","Data":"7c5361a6e26962ec8e0a4804e8c569f5cf32e0661016cf4a9a2740fc52fe7d46"}
Nov 03 20:10:17.927257 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:17.927133    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"
Nov 03 20:10:17.964508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:17.962006    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst" podStartSLOduration=1.961982978 podStartE2EDuration="1.961982978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:17.947714645 +0000 UTC m=+1970.102617071" watchObservedRunningTime="2026-11-03 20:10:17.961982978 +0000 UTC m=+1970.116885397"
Nov 03 20:10:18.076560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:18.073180    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:10:18.076560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:18.073243    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7" podUID="9a68390b-ddc0-4718-856b-495251cf9647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 20:10:18.206505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:18.204884    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:18.926348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:18.926317    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:10:18.926612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:18.926588    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" podUID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:10:19.067914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:19.067874    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:10:19.068224 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:19.068195    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7" podUID="9a68390b-ddc0-4718-856b-495251cf9647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 20:10:19.323063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:19.323004    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:19.886065 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:19.885980    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:19.925880 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:19.925845    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:10:19.926052 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:19.925903    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" podUID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:10:20.069946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:20.069906    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 03 20:10:20.070237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:20.070207    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7" podUID="9a68390b-ddc0-4718-856b-495251cf9647" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 03 20:10:20.244099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:20.244064    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:20.925778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:20.925737    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:10:20.926263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:20.925797    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" podUID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:10:20.986955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:20.986905    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"]
Nov 03 20:10:20.994087 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:20.994028    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.001301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.000023    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0e4ce7-serving-cert\""
Nov 03 20:10:21.010198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.010165    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"]
Nov 03 20:10:21.071166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.071127    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:21.079197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.079168    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:21.082561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.080362    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.082561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.080471    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.082561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.080535    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-metrics-certs\") pod \"maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.082561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.080638    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hcrp\" (UniqueName: \"kubernetes.io/projected/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-kube-api-access-9hcrp\") pod \"maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.082561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.080793    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:21.089008 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.088975    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:10:21.192964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.191823    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.192964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.192011    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.192964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.192069    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-metrics-certs\") pod \"maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.192964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.192231    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hcrp\" (UniqueName: \"kubernetes.io/projected/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-kube-api-access-9hcrp\") pod \"maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.193308 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:10:21.192974    5659 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-serving-cert: secret "maas-api-e2e-extmod-b-0e4ce7-serving-cert" not found
Nov 03 20:10:21.193308 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:10:21.193039    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-maas-api-tls podName:bd928a60-b898-49f3-8ff0-5dfc52ef4fe4 nodeName:}" failed. No retries permitted until 2026-11-03 20:10:21.693019184 +0000 UTC m=+1973.847921583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-maas-api-tls") pod "maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc" (UID: "bd928a60-b898-49f3-8ff0-5dfc52ef4fe4") : secret "maas-api-e2e-extmod-b-0e4ce7-serving-cert" not found
Nov 03 20:10:21.194162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.194132    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.199104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.198149    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-metrics-certs\") pod \"maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.223425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.223360    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hcrp\" (UniqueName: \"kubernetes.io/projected/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-kube-api-access-9hcrp\") pod \"maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.229664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.226386    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:21.708262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.708220    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.710914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.710885    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.739508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.737973    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:21.930506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.929565    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:21.930506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.930162    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:10:21.930506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:21.930211    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" podUID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:10:22.265621 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:22.264781    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:22.348751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:22.347632    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"]
Nov 03 20:10:22.724058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:22.724029    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:22.865875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:22.865836    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:22.932166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:22.932142    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:22.932730 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:22.932704    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:22.933671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:22.933649    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:10:23.049604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:23.049464    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc" event={"ID":"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4","Type":"ContainerStarted","Data":"4a5a3154694c8844038d3f8dab150660417870576819520f01d423aeb99731d2"}
Nov 03 20:10:23.049604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:23.049572    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc" event={"ID":"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4","Type":"ContainerStarted","Data":"6de54aa10b8e6f7a37afb9fe46348916c82126d4b6a2278fb2b3f94c81948924"}
Nov 03 20:10:23.050531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:23.050502    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:23.061281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:23.061237    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:23.078450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:23.078386    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc" podStartSLOduration=2.576349607 podStartE2EDuration="3.078365619s" totalImagesPullingTime="502.016012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:23.075619559 +0000 UTC m=+1975.230521989" watchObservedRunningTime="2026-11-03 20:10:23.078365619 +0000 UTC m=+1975.233268041"
Nov 03 20:10:23.201511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:23.201126    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:23.345269 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:23.345174    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:23.518661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:23.518625    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:23.666798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:23.666705    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:23.995090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:23.990008    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:24.186498 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.186449    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:24.341525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.341415    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:24.866234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.866184    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:24.871166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.871135    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"]
Nov 03 20:10:24.875873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.875849    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:24.888117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.888083    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0e1546\""
Nov 03 20:10:24.889244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.889218    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"kube-root-ca.crt\""
Nov 03 20:10:24.889244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.889234    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"routing-overlay\""
Nov 03 20:10:24.889244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.889221    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"openshift-service-ca.crt\""
Nov 03 20:10:24.889579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.889563    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"agc-cred-aff1b5d2e147\""
Nov 03 20:10:24.889653 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.889631    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"payload-processing-external-model-e2e--4cc23329-dockercfg-54nwd\""
Nov 03 20:10:24.892059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.892032    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"]
Nov 03 20:10:24.927621 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.927581    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:24.927808 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.927779    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6cnr\" (UniqueName: \"kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-kube-api-access-v6cnr\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:24.927926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.927900    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:24.928004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:24.927988    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:25.030078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:25.030040    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:25.030578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:25.030165    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6cnr\" (UniqueName: \"kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-kube-api-access-v6cnr\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:25.030578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:25.030214    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:25.030578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:25.030257    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:25.030885 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:25.030860    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:25.031092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:25.031068    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:25.033013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:25.032985    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:25.048190 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:25.048152    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6cnr\" (UniqueName: \"kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-kube-api-access-v6cnr\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:25.196557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:25.195850    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:25.235506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:25.235077    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:25.667840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:25.667804    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"]
Nov 03 20:10:25.668129 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:10:25.668093    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a937b1_fc8f_41ed_8cc8_ef54e20bae06.slice/crio-7bb2e499c451624feb15195c6e78ec21e930325a16486a5e7eaaad7df79c09b8 WatchSource:0}: Error finding container 7bb2e499c451624feb15195c6e78ec21e930325a16486a5e7eaaad7df79c09b8: Status 404 returned error can't find the container with id 7bb2e499c451624feb15195c6e78ec21e930325a16486a5e7eaaad7df79c09b8
Nov 03 20:10:26.130843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:26.130653    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:26.198565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:26.197805    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn" event={"ID":"03a937b1-fc8f-41ed-8cc8-ef54e20bae06","Type":"ContainerStarted","Data":"0d8c6078b5dc6c23f9414f493bf8d985bb3fc61e2488f90b1bbf4ce3c5c44e69"}
Nov 03 20:10:26.198565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:26.197867    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn" event={"ID":"03a937b1-fc8f-41ed-8cc8-ef54e20bae06","Type":"ContainerStarted","Data":"7bb2e499c451624feb15195c6e78ec21e930325a16486a5e7eaaad7df79c09b8"}
Nov 03 20:10:26.198565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:26.197909    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:26.220740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:26.220689    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn" podStartSLOduration=2.220675788 podStartE2EDuration="2.220675788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:26.219441651 +0000 UTC m=+1978.374344071" watchObservedRunningTime="2026-11-03 20:10:26.220675788 +0000 UTC m=+1978.375578206"
Nov 03 20:10:26.839539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:26.839475    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:27.958539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:27.958467    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:29.016761 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:29.016730    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"
Nov 03 20:10:29.110921 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:29.110888    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:29.904467 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:29.904434    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:30.149322 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:30.149279    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:10:30.695978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:30.690119    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:31.440973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:31.440937    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:32.132400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:32.132352    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:32.867644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:32.867599    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:33.543765 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:33.543734    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:34.232780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:34.232749    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:34.953352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:34.953300    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:35.770130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:35.770059    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:35.921529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:35.921467    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:36.568359 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:36.568322    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:37.159516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:37.159447    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:37.208713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:37.208688    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:10:37.481150 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:37.481103    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:38.288803 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:38.288768    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:38.934286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:38.934195    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:39.682504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:39.679325    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:40.344737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:40.344706    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:41.054767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:41.054733    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:41.861039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:41.861004    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:42.777502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:42.777460    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:43.545938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:43.545908    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:44.213251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:44.213215    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:44.968468 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:44.968432    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:45.218317 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:10:45.218274    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:10:45.800102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:45.800066    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:45.933894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:45.933861    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:46.462990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:46.462954    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:47.094432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:47.094392    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:47.284582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:47.284547    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:47.427214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:47.427140    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:47.576271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:47.576236    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:47.748235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:47.748196    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:47.887110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:47.887076    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:48.459049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:48.459019    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:49.153878 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:49.153845    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:49.832715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:49.832680    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:50.470611 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:50.470581    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:51.141986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:51.141949    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:51.778961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:51.778924    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:52.458511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:52.458465    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:53.154124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:53.154092    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:53.829502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:53.829460    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:54.584698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:54.584659    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:55.596919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:55.596877    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:56.366151 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:56.366121    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:57.374826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.374785    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:57.786514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.786420    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:57.818684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.818645    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"]
Nov 03 20:10:57.838197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.838166    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:57.838977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.838413    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"]
Nov 03 20:10:57.904671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.904635    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6slj5\" (UniqueName: \"kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-kube-api-access-6slj5\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:57.904837 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.904679    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:57.904887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.904840    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:57.904931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.904898    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:57.904968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.904952    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:57.905193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.905125    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:57.905302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.905234    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:57.905302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.905278    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:57.905527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.905502    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:57.938841 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:57.938814    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:58.007007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.006960    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.007202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.007040    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.007202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.007086    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.007202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.007172    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.007202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.007199    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.007452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.007228    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.007452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.007309    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.007452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.007374    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6slj5\" (UniqueName: \"kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-kube-api-access-6slj5\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.007452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.007405    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.007727 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.007627    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.007864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.007841    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.008021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.007869    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.008125 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.007884    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.008188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.008069    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.009601 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.009576    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.010383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.010361    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.022727 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.022702    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.023307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.023288    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6slj5\" (UniqueName: \"kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-kube-api-access-6slj5\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-967xd\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.165078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.163803    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:10:58.584106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.583931    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:58.715623 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.714641    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"]
Nov 03 20:10:58.736382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.732417    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"]
Nov 03 20:10:58.738166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.738113    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:58.786867 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.786828    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:58.787901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.787844    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"]
Nov 03 20:10:58.790664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.790112    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:58.790664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.790193    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:58.790664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.790236    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:58.997110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:58.997081    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:59.022509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.007548    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.022509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.007601    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.022509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.007664    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.022509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.007736    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.022509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.007791    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.022509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.007969    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.022509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.008093    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.022509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.008209    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s8tf\" (UniqueName: \"kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-kube-api-access-5s8tf\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.022509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.008259    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.120648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.115394    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.120648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.115523    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s8tf\" (UniqueName: \"kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-kube-api-access-5s8tf\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.120648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.115567    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.120648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.115655    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.120648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.115691    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.120648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.115744    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.120648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.115811    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.120648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.115856    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.120648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.116024    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.120648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.117938    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.120648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.118199    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.135414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.133984    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.135414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.134436    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.135414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.134705    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.170950 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.170825    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.184339 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.176539    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s8tf\" (UniqueName: \"kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-kube-api-access-5s8tf\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.192061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.191984    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.245301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.223116    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.315474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.303375    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:59.484254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.484222    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:10:59.918682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.918580    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd" event={"ID":"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a","Type":"ContainerStarted","Data":"500432627ccc158c19fdef9c1f5c2e1d1f087237ebed5309e5d8cbcf053844de"}
Nov 03 20:10:59.920452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:10:59.918689    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd" event={"ID":"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a","Type":"ContainerStarted","Data":"bef1dfd2cd351d8a4802548071187c67abf4b9f3290f4d2b716a40eb6211d313"}
Nov 03 20:11:00.160172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:00.160116    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd" podStartSLOduration=3.16010092 podStartE2EDuration="3.16010092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:59.949301339 +0000 UTC m=+2012.104203790" watchObservedRunningTime="2026-11-03 20:11:00.16010092 +0000 UTC m=+2012.315003340"
Nov 03 20:11:00.164802 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:00.163529    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"]
Nov 03 20:11:00.167641 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:00.166591    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:11:00.168034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:00.168009    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-967xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:11:00.168121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:00.168062    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd" podUID="296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 20:11:00.170308 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:00.170269    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:00.170422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:00.170360    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:00.170422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:00.170404    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:00.609508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:00.596046    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:00.745507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:00.738024    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:00.940558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:00.931761    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs" event={"ID":"2477710a-1350-4fa2-9381-b2e6c4fcb6b1","Type":"ContainerStarted","Data":"423aaa21382ab1445b77bee899471fd72d3834e5d35faaf174ae80d5574871bf"}
Nov 03 20:11:00.940558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:00.931829    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs" event={"ID":"2477710a-1350-4fa2-9381-b2e6c4fcb6b1","Type":"ContainerStarted","Data":"63d546880b4cf72f0ce1379e646a3c398ab283784efe688b98c35659eca2d36c"}
Nov 03 20:11:01.008375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:01.008309    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs" podStartSLOduration=3.008288452 podStartE2EDuration="3.008288452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:00.98262699 +0000 UTC m=+2013.137529413" watchObservedRunningTime="2026-11-03 20:11:01.008288452 +0000 UTC m=+2013.163190872"
Nov 03 20:11:01.182505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:01.181723    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-967xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:11:01.182505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:01.181785    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd" podUID="296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 20:11:01.491544 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:01.489317    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:11:01.491544 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:01.489460    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 20:11:01.491544 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:01.489528    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs" podUID="2477710a-1350-4fa2-9381-b2e6c4fcb6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 20:11:01.763564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:01.762992    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:02.010656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:02.009441    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:02.175452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:02.175367    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:02.183975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:02.178909    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-967xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:11:02.184312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:02.184259    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd" podUID="296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 20:11:02.365512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:02.364882    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:02.491536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:02.487751    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 20:11:02.491536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:02.487809    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs" podUID="2477710a-1350-4fa2-9381-b2e6c4fcb6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 20:11:03.170518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:03.169977    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-967xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:11:03.170518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:03.170039    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd" podUID="296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 20:11:03.291522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:03.288105    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:03.489650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:03.489608    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 20:11:03.489866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:03.489676    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs" podUID="2477710a-1350-4fa2-9381-b2e6c4fcb6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 20:11:03.895938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:03.895819    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:04.169091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:04.168987    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-967xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:11:04.169375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:04.169337    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd" podUID="296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 20:11:04.485973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:04.485932    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 20:11:04.486647 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:04.486613    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs" podUID="2477710a-1350-4fa2-9381-b2e6c4fcb6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 20:11:04.600896 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:04.600809    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:05.165465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:05.165033    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-967xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:11:05.165465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:05.165096    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd" podUID="296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 20:11:05.487648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:05.487596    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 20:11:05.488171 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:05.487676    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs" podUID="2477710a-1350-4fa2-9381-b2e6c4fcb6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 20:11:05.854952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:05.854865    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:06.183565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:06.166223    5659 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-967xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:11:06.183869 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:06.183822    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd" podUID="296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 20:11:06.487158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:06.484978    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 20:11:06.487158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:06.485038    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs" podUID="2477710a-1350-4fa2-9381-b2e6c4fcb6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 20:11:06.868119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:06.867996    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:07.170340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:07.170246    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:11:07.170820 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:07.170798    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:11:07.171611 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:07.171588    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:11:07.493346 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:07.493309    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:11:07.493780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:07.493755    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:11:07.494754 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:07.494729    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:11:07.729078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:07.729042    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:08.396403 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:08.396367    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:09.166789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:09.166752    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:09.928249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:09.928218    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:10.645042 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:10.644987    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:11.332983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:11.332944    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:12.118208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:12.118172    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:12.507690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:12.507653    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:12.654738 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:12.654708    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:12.957971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:12.957924    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:13.187104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:13.187058    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:13.321972 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:13.321890    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:14.264557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:14.263837    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:14.389271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:14.389233    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:15.219019 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:11:15.218981    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:11:15.233117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:15.233084    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:16.053724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:16.053686    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:16.884183 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:16.884115    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:17.791404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:17.791329    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:18.872594 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:18.872553    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:19.795505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:19.791525    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:20.707099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:20.707059    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:21.303917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:21.303882    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:21.483451 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:21.483417    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:21.623124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:21.623023    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:22.066434 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:22.066389    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:22.226569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:22.214985    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:22.378869 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:22.378764    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:22.553702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:22.553661    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:23.124699 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:23.124660    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:23.260356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:23.260318    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:23.869740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:23.869703    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:24.022512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:24.020826    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:24.548461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:24.548409    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:25.437281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:25.437246    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:26.228881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:26.228850    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:27.099625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:27.099588    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:27.762452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:27.762415    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:28.494234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:28.494198    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:29.908296 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:29.908258    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"]
Nov 03 20:11:29.908817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:29.908542    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7" podUID="9a68390b-ddc0-4718-856b-495251cf9647" containerName="istio-proxy" containerID="cri-o://01c0d077cc105a45b32b8af691c352a81b13b88d9ef946868917b9ebb630220e" gracePeriod=30
Nov 03 20:11:29.919110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:29.919084    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"]
Nov 03 20:11:29.919298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:29.919277    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd" podUID="296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" containerName="istio-proxy" containerID="cri-o://500432627ccc158c19fdef9c1f5c2e1d1f087237ebed5309e5d8cbcf053844de" gracePeriod=30
Nov 03 20:11:30.065504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:30.063037    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:30.637048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:30.636981    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:31.345805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:31.345764    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:31.813956 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:31.813918    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:32.046960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:32.046919    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:32.304598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:32.304564    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:32.478378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:32.478309    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:33.029298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:33.029268    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:34.065679 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:34.065640    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:34.999902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:34.999872    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:35.279524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.276849    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:11:35.343254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.337300    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:11:35.375438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.372539    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-socket\") pod \"9a68390b-ddc0-4718-856b-495251cf9647\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") "
Nov 03 20:11:35.375438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.372687    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a68390b-ddc0-4718-856b-495251cf9647-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a68390b-ddc0-4718-856b-495251cf9647-istio-podinfo\") pod \"9a68390b-ddc0-4718-856b-495251cf9647\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") "
Nov 03 20:11:35.375438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.372810    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-credential-socket\") pod \"9a68390b-ddc0-4718-856b-495251cf9647\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") "
Nov 03 20:11:35.375438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.372859    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-certs\") pod \"9a68390b-ddc0-4718-856b-495251cf9647\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") "
Nov 03 20:11:35.375438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.372890    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-data\") pod \"9a68390b-ddc0-4718-856b-495251cf9647\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") "
Nov 03 20:11:35.375438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.372955    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-kube-api-access-p9ffv\" (UniqueName: \"kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-kube-api-access-p9ffv\") pod \"9a68390b-ddc0-4718-856b-495251cf9647\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") "
Nov 03 20:11:35.375438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.373098    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-istio-token\" (UniqueName: \"kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-istio-token\") pod \"9a68390b-ddc0-4718-856b-495251cf9647\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") "
Nov 03 20:11:35.375438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.373128    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a68390b-ddc0-4718-856b-495251cf9647-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a68390b-ddc0-4718-856b-495251cf9647-istiod-ca-cert\") pod \"9a68390b-ddc0-4718-856b-495251cf9647\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") "
Nov 03 20:11:35.375438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.373224    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-envoy\") pod \"9a68390b-ddc0-4718-856b-495251cf9647\" (UID: \"9a68390b-ddc0-4718-856b-495251cf9647\") "
Nov 03 20:11:35.375438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.373994    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-certs" pod "9a68390b-ddc0-4718-856b-495251cf9647" (UID: "9a68390b-ddc0-4718-856b-495251cf9647"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.375438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.374537    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-socket" pod "9a68390b-ddc0-4718-856b-495251cf9647" (UID: "9a68390b-ddc0-4718-856b-495251cf9647"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.375438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.374831    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-credential-socket" pod "9a68390b-ddc0-4718-856b-495251cf9647" (UID: "9a68390b-ddc0-4718-856b-495251cf9647"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.375438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.375325    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a68390b-ddc0-4718-856b-495251cf9647-istiod-ca-cert" pod "9a68390b-ddc0-4718-856b-495251cf9647" (UID: "9a68390b-ddc0-4718-856b-495251cf9647"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:35.376429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.376382    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-data" pod "9a68390b-ddc0-4718-856b-495251cf9647" (UID: "9a68390b-ddc0-4718-856b-495251cf9647"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.378137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.378106    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-istio-token" pod "9a68390b-ddc0-4718-856b-495251cf9647" (UID: "9a68390b-ddc0-4718-856b-495251cf9647"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:35.382748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.382710    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-kube-api-access-p9ffv" pod "9a68390b-ddc0-4718-856b-495251cf9647" (UID: "9a68390b-ddc0-4718-856b-495251cf9647"). InnerVolumeSpecName "kube-api-access-p9ffv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:35.389328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.388191    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a68390b-ddc0-4718-856b-495251cf9647-istio-podinfo" pod "9a68390b-ddc0-4718-856b-495251cf9647" (UID: "9a68390b-ddc0-4718-856b-495251cf9647"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:11:35.389328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.388267    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-envoy" pod "9a68390b-ddc0-4718-856b-495251cf9647" (UID: "9a68390b-ddc0-4718-856b-495251cf9647"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.475679 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.475631    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-socket\") pod \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") "
Nov 03 20:11:35.475893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.475694    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-token\" (UniqueName: \"kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-token\") pod \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") "
Nov 03 20:11:35.475893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.475796    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-envoy\") pod \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") "
Nov 03 20:11:35.475893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.475865    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-credential-socket\") pod \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") "
Nov 03 20:11:35.476076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.475954    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-podinfo\") pod \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") "
Nov 03 20:11:35.476076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.475961    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-socket" pod "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" (UID: "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.476076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.475996    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-kube-api-access-6slj5\" (UniqueName: \"kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-kube-api-access-6slj5\") pod \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") "
Nov 03 20:11:35.476076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.476058    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istiod-ca-cert\") pod \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") "
Nov 03 20:11:35.476284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.476229    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-certs\") pod \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") "
Nov 03 20:11:35.476284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.476280    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-data\") pod \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\" (UID: \"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a\") "
Nov 03 20:11:35.477382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.477047    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.477382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.477073    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.477382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.477088    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.477382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.477103    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9ffv\" (UniqueName: \"kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-kube-api-access-p9ffv\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.477382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.477117    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a68390b-ddc0-4718-856b-495251cf9647-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.477382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.477131    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a68390b-ddc0-4718-856b-495251cf9647-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.477382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.477145    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.477382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.477158    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a68390b-ddc0-4718-856b-495251cf9647-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.477382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.477171    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.477382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.477185    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a68390b-ddc0-4718-856b-495251cf9647-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.477992 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.477400    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-credential-socket" pod "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" (UID: "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.477992 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.477468    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-certs" pod "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" (UID: "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.477992 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.477927    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istiod-ca-cert" pod "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" (UID: "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:35.478240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.478205    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-data" pod "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" (UID: "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.481259 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.479110    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-envoy" pod "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" (UID: "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.481657 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.481318    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-token" pod "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" (UID: "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:35.482765 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.482729    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-podinfo" pod "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" (UID: "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:11:35.483920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.483892    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-kube-api-access-6slj5" pod "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" (UID: "296e6c65-1ea7-47eb-bdd4-f97fdd03e15a"). InnerVolumeSpecName "kube-api-access-6slj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:35.495498 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.495462    5659 generic.go:356] "Generic (PLEG): container finished" podID="296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" containerID="500432627ccc158c19fdef9c1f5c2e1d1f087237ebed5309e5d8cbcf053844de" exitCode=0
Nov 03 20:11:35.495629 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.495514    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd" event={"ID":"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a","Type":"ContainerDied","Data":"500432627ccc158c19fdef9c1f5c2e1d1f087237ebed5309e5d8cbcf053844de"}
Nov 03 20:11:35.495629 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.495558    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"
Nov 03 20:11:35.495629 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.495580    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd" event={"ID":"296e6c65-1ea7-47eb-bdd4-f97fdd03e15a","Type":"ContainerDied","Data":"bef1dfd2cd351d8a4802548071187c67abf4b9f3290f4d2b716a40eb6211d313"}
Nov 03 20:11:35.495629 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.495611    5659 scope.go:122] "RemoveContainer" containerID="500432627ccc158c19fdef9c1f5c2e1d1f087237ebed5309e5d8cbcf053844de"
Nov 03 20:11:35.497139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.497052    5659 generic.go:356] "Generic (PLEG): container finished" podID="9a68390b-ddc0-4718-856b-495251cf9647" containerID="01c0d077cc105a45b32b8af691c352a81b13b88d9ef946868917b9ebb630220e" exitCode=0
Nov 03 20:11:35.497207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.497183    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7" event={"ID":"9a68390b-ddc0-4718-856b-495251cf9647","Type":"ContainerDied","Data":"01c0d077cc105a45b32b8af691c352a81b13b88d9ef946868917b9ebb630220e"}
Nov 03 20:11:35.497278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.497262    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7" event={"ID":"9a68390b-ddc0-4718-856b-495251cf9647","Type":"ContainerDied","Data":"df3fe513cefb4c53109a33f763495baf3851f698600a79e20167a630d4fb7258"}
Nov 03 20:11:35.497351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.497340    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"
Nov 03 20:11:35.510433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.510415    5659 scope.go:122] "RemoveContainer" containerID="500432627ccc158c19fdef9c1f5c2e1d1f087237ebed5309e5d8cbcf053844de"
Nov 03 20:11:35.510961 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:11:35.510926    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500432627ccc158c19fdef9c1f5c2e1d1f087237ebed5309e5d8cbcf053844de\": container with ID starting with 500432627ccc158c19fdef9c1f5c2e1d1f087237ebed5309e5d8cbcf053844de not found: ID does not exist" containerID="500432627ccc158c19fdef9c1f5c2e1d1f087237ebed5309e5d8cbcf053844de"
Nov 03 20:11:35.511056 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.510964    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500432627ccc158c19fdef9c1f5c2e1d1f087237ebed5309e5d8cbcf053844de"} err="failed to get container status \"500432627ccc158c19fdef9c1f5c2e1d1f087237ebed5309e5d8cbcf053844de\": rpc error: code = NotFound desc = could not find container \"500432627ccc158c19fdef9c1f5c2e1d1f087237ebed5309e5d8cbcf053844de\": container with ID starting with 500432627ccc158c19fdef9c1f5c2e1d1f087237ebed5309e5d8cbcf053844de not found: ID does not exist"
Nov 03 20:11:35.511056 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.510986    5659 scope.go:122] "RemoveContainer" containerID="01c0d077cc105a45b32b8af691c352a81b13b88d9ef946868917b9ebb630220e"
Nov 03 20:11:35.523993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.523974    5659 scope.go:122] "RemoveContainer" containerID="01c0d077cc105a45b32b8af691c352a81b13b88d9ef946868917b9ebb630220e"
Nov 03 20:11:35.524302 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:11:35.524281    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c0d077cc105a45b32b8af691c352a81b13b88d9ef946868917b9ebb630220e\": container with ID starting with 01c0d077cc105a45b32b8af691c352a81b13b88d9ef946868917b9ebb630220e not found: ID does not exist" containerID="01c0d077cc105a45b32b8af691c352a81b13b88d9ef946868917b9ebb630220e"
Nov 03 20:11:35.524372 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.524314    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c0d077cc105a45b32b8af691c352a81b13b88d9ef946868917b9ebb630220e"} err="failed to get container status \"01c0d077cc105a45b32b8af691c352a81b13b88d9ef946868917b9ebb630220e\": rpc error: code = NotFound desc = could not find container \"01c0d077cc105a45b32b8af691c352a81b13b88d9ef946868917b9ebb630220e\": container with ID starting with 01c0d077cc105a45b32b8af691c352a81b13b88d9ef946868917b9ebb630220e not found: ID does not exist"
Nov 03 20:11:35.579756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.579715    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.579756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.579744    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.579756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.579760    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6slj5\" (UniqueName: \"kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-kube-api-access-6slj5\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.579995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.579773    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.579995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.579785    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.579995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.579801    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.579995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.579812    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.579995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.579823    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.586105 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.586074    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"]
Nov 03 20:11:35.608136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.608090    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-967xd"]
Nov 03 20:11:35.660170 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.660118    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"]
Nov 03 20:11:35.665236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.665197    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-v6sm7"]
Nov 03 20:11:35.770017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.769909    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:35.894072 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:35.894038    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:36.030560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:36.030064    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:36.059505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:36.059239    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" path="/var/lib/kubelet/pods/296e6c65-1ea7-47eb-bdd4-f97fdd03e15a/volumes"
Nov 03 20:11:36.076505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:36.073247    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a68390b-ddc0-4718-856b-495251cf9647" path="/var/lib/kubelet/pods/9a68390b-ddc0-4718-856b-495251cf9647/volumes"
Nov 03 20:11:36.871506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:36.867606    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:37.152375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.152269    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:37.203547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.203515    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"]
Nov 03 20:11:37.204201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.204187    5659 cpu_manager.go:414] "containerMap: removing container" podUID="296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" containerName="istio-proxy"
Nov 03 20:11:37.204266 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.204202    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" containerName="istio-proxy"
Nov 03 20:11:37.204266 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.204211    5659 cpu_manager.go:414] "containerMap: removing container" podUID="9a68390b-ddc0-4718-856b-495251cf9647" containerName="istio-proxy"
Nov 03 20:11:37.204266 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.204216    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a68390b-ddc0-4718-856b-495251cf9647" containerName="istio-proxy"
Nov 03 20:11:37.204367 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.204346    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="296e6c65-1ea7-47eb-bdd4-f97fdd03e15a" containerName="istio-proxy"
Nov 03 20:11:37.204367 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.204355    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a68390b-ddc0-4718-856b-495251cf9647" containerName="istio-proxy"
Nov 03 20:11:37.209607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.209586    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"
Nov 03 20:11:37.216642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.216617    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"]
Nov 03 20:11:37.217368 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.217338    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-33e267-dockercfg-z47nh\""
Nov 03 20:11:37.218604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.218582    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-33e267\""
Nov 03 20:11:37.221956 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.221935    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"]
Nov 03 20:11:37.245612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.245565    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"]
Nov 03 20:11:37.245796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.245700    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"
Nov 03 20:11:37.266850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.265588    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw8xd\" (UniqueName: \"kubernetes.io/projected/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-kube-api-access-xw8xd\") pod \"payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58\" (UID: \"bfcbe9c0-443c-4551-93f9-1b50010ff9f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"
Nov 03 20:11:37.266850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.265631    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58\" (UID: \"bfcbe9c0-443c-4551-93f9-1b50010ff9f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"
Nov 03 20:11:37.336197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.336162    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:37.390508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.368626    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw8xd\" (UniqueName: \"kubernetes.io/projected/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-kube-api-access-xw8xd\") pod \"payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58\" (UID: \"bfcbe9c0-443c-4551-93f9-1b50010ff9f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"
Nov 03 20:11:37.390508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.368684    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58\" (UID: \"bfcbe9c0-443c-4551-93f9-1b50010ff9f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"
Nov 03 20:11:37.390508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.368767    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7356100-2a57-4289-b38f-133d56fa44d1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd\" (UID: \"a7356100-2a57-4289-b38f-133d56fa44d1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"
Nov 03 20:11:37.390508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.368969    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95zbk\" (UniqueName: \"kubernetes.io/projected/a7356100-2a57-4289-b38f-133d56fa44d1-kube-api-access-95zbk\") pod \"payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd\" (UID: \"a7356100-2a57-4289-b38f-133d56fa44d1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"
Nov 03 20:11:37.390508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.370297    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58\" (UID: \"bfcbe9c0-443c-4551-93f9-1b50010ff9f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"
Nov 03 20:11:37.491981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.491942    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95zbk\" (UniqueName: \"kubernetes.io/projected/a7356100-2a57-4289-b38f-133d56fa44d1-kube-api-access-95zbk\") pod \"payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd\" (UID: \"a7356100-2a57-4289-b38f-133d56fa44d1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"
Nov 03 20:11:37.492212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.492191    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7356100-2a57-4289-b38f-133d56fa44d1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd\" (UID: \"a7356100-2a57-4289-b38f-133d56fa44d1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"
Nov 03 20:11:37.546984 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.539777    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw8xd\" (UniqueName: \"kubernetes.io/projected/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-kube-api-access-xw8xd\") pod \"payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58\" (UID: \"bfcbe9c0-443c-4551-93f9-1b50010ff9f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"
Nov 03 20:11:37.574627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.564830    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"
Nov 03 20:11:37.610127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.610086    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7356100-2a57-4289-b38f-133d56fa44d1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd\" (UID: \"a7356100-2a57-4289-b38f-133d56fa44d1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"
Nov 03 20:11:37.624574 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.624534    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95zbk\" (UniqueName: \"kubernetes.io/projected/a7356100-2a57-4289-b38f-133d56fa44d1-kube-api-access-95zbk\") pod \"payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd\" (UID: \"a7356100-2a57-4289-b38f-133d56fa44d1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"
Nov 03 20:11:37.873709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.873615    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"
Nov 03 20:11:37.926994 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:11:37.926178    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcbe9c0_443c_4551_93f9_1b50010ff9f0.slice/crio-a820025b1428133936e84bce9467b6bd745304b5b081afd5ac7959c5d85ffc35 WatchSource:0}: Error finding container a820025b1428133936e84bce9467b6bd745304b5b081afd5ac7959c5d85ffc35: Status 404 returned error can't find the container with id a820025b1428133936e84bce9467b6bd745304b5b081afd5ac7959c5d85ffc35
Nov 03 20:11:37.927543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:37.927472    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"]
Nov 03 20:11:38.064858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:38.064790    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:38.175216 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:11:38.175188    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7356100_2a57_4289_b38f_133d56fa44d1.slice/crio-5caccd540619dcf81cadf3473052e6507eb1506af1b0b57b9583c9cd7254a7d1 WatchSource:0}: Error finding container 5caccd540619dcf81cadf3473052e6507eb1506af1b0b57b9583c9cd7254a7d1: Status 404 returned error can't find the container with id 5caccd540619dcf81cadf3473052e6507eb1506af1b0b57b9583c9cd7254a7d1
Nov 03 20:11:38.175462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:38.175440    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"]
Nov 03 20:11:38.637628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:38.637590    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58" event={"ID":"bfcbe9c0-443c-4551-93f9-1b50010ff9f0","Type":"ContainerStarted","Data":"ae9e88d90a6c9997803e33594a2c4fdcc640016ef80923217df8ff6fcc6fa0db"}
Nov 03 20:11:38.637861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:38.637685    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58" event={"ID":"bfcbe9c0-443c-4551-93f9-1b50010ff9f0","Type":"ContainerStarted","Data":"a820025b1428133936e84bce9467b6bd745304b5b081afd5ac7959c5d85ffc35"}
Nov 03 20:11:38.637861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:38.637831    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"
Nov 03 20:11:38.639885 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:38.639860    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd" event={"ID":"a7356100-2a57-4289-b38f-133d56fa44d1","Type":"ContainerStarted","Data":"a61626e49efa7334c7e4ab289ee564a4b49614c796efd0a4f04bd466ef945502"}
Nov 03 20:11:38.640006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:38.639907    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd" event={"ID":"a7356100-2a57-4289-b38f-133d56fa44d1","Type":"ContainerStarted","Data":"5caccd540619dcf81cadf3473052e6507eb1506af1b0b57b9583c9cd7254a7d1"}
Nov 03 20:11:38.640006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:38.639936    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"
Nov 03 20:11:38.660110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:38.660057    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58" podStartSLOduration=1.6600407879999999 podStartE2EDuration="1.660040788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:38.655703918 +0000 UTC m=+2050.810606339" watchObservedRunningTime="2026-11-03 20:11:38.660040788 +0000 UTC m=+2050.814943207"
Nov 03 20:11:38.677974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:38.677916    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd" podStartSLOduration=1.67789858 podStartE2EDuration="1.67789858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:38.674027609 +0000 UTC m=+2050.828930028" watchObservedRunningTime="2026-11-03 20:11:38.67789858 +0000 UTC m=+2050.832801001"
Nov 03 20:11:38.857455 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:38.857409    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:39.555964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:39.555923    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:40.253529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:40.253474    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:40.943137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:40.943087    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:41.766037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:41.765961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:42.754956 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:42.754926    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:43.629533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:43.629500    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:44.338701 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:44.338666    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:45.197377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:45.197344    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:45.265250 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:11:45.265212    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:11:45.398122 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:45.398093    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:45.530807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:45.530774    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:45.691142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:45.691107    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:46.168145 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:46.167599    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:46.301518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:46.301449    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:46.927821 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:46.927785    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"]
Nov 03 20:11:46.932942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:46.932917    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:46.939967 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:46.939940    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"agc-cred-1db7c4076920\""
Nov 03 20:11:46.942635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:46.942607    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"]
Nov 03 20:11:47.099260 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.099224    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.099418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.099349    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.099512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.099437    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.099701 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.099682    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dld94\" (UniqueName: \"kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-kube-api-access-dld94\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.201169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.201089    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.201169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.201152    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.201396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.201204    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.201396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.201362    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dld94\" (UniqueName: \"kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-kube-api-access-dld94\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.201890 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.201867    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.201890 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.201882    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.203746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.203725    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.213442 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.213417    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dld94\" (UniqueName: \"kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-kube-api-access-dld94\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.255242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.255214    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.361708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.334141    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:47.542315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.542276    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"]
Nov 03 20:11:47.553474 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:11:47.553420    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0a30ee_0be2_4abf_8624_d76f418d104f.slice/crio-652608ae475a22b22223c3d5596d0c0cc14c0ebddf98d3bf648e221334d35a26 WatchSource:0}: Error finding container 652608ae475a22b22223c3d5596d0c0cc14c0ebddf98d3bf648e221334d35a26: Status 404 returned error can't find the container with id 652608ae475a22b22223c3d5596d0c0cc14c0ebddf98d3bf648e221334d35a26
Nov 03 20:11:47.757324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.757290    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2" event={"ID":"9f0a30ee-0be2-4abf-8624-d76f418d104f","Type":"ContainerStarted","Data":"2e2d472a12057d9aa721fa893abf8db175995e2d4529ba61107c898d55e9e90d"}
Nov 03 20:11:47.757592 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.757355    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2" event={"ID":"9f0a30ee-0be2-4abf-8624-d76f418d104f","Type":"ContainerStarted","Data":"652608ae475a22b22223c3d5596d0c0cc14c0ebddf98d3bf648e221334d35a26"}
Nov 03 20:11:47.757592 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.757385    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:47.778391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:47.778333    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2" podStartSLOduration=1.778318927 podStartE2EDuration="1.778318927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:47.775999629 +0000 UTC m=+2059.930902070" watchObservedRunningTime="2026-11-03 20:11:47.778318927 +0000 UTC m=+2059.933221346"
Nov 03 20:11:48.054556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:48.054521    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:48.836452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:48.836415    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:49.561178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:49.561141    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:49.653426 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:49.653392    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"
Nov 03 20:11:49.653653 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:49.653473    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"
Nov 03 20:11:50.332923 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:50.332889    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:51.030703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:51.030671    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:51.830163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:51.830134    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:52.602937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:52.602901    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:53.446321 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:53.446287    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:54.146987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:54.146951    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:54.929094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:54.929057    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:55.075580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:55.069385    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:55.837835 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:55.837790    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:56.587291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:56.587255    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:57.543334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:57.543292    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:58.381503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:58.378400    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:58.769018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:58.768987    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:11:58.829704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:58.829659    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"]
Nov 03 20:11:59.039858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:59.039752    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:59.814230 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:11:59.814193    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:00.518692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:00.518657    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:01.330201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:01.330169    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:02.033333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:02.033299    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:02.729252 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:02.729199    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:03.456440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:03.456388    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:03.830888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:03.830769    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn" podUID="03a937b1-fc8f-41ed-8cc8-ef54e20bae06" containerName="payload-processing" containerID="cri-o://0d8c6078b5dc6c23f9414f493bf8d985bb3fc61e2488f90b1bbf4ce3c5c44e69" gracePeriod=25
Nov 03 20:12:04.128576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.128544    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:04.132309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.132288    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:12:04.287873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.287840    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-provider-credentials\" (UniqueName: \"kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-provider-credentials\") pod \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") "
Nov 03 20:12:04.287873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.287883    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-routing-overlay-volume\") pod \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") "
Nov 03 20:12:04.288169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.288062    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-kube-api-access-v6cnr\" (UniqueName: \"kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-kube-api-access-v6cnr\") pod \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") "
Nov 03 20:12:04.288169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.288081    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-plugins-config-volume\") pod \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\" (UID: \"03a937b1-fc8f-41ed-8cc8-ef54e20bae06\") "
Nov 03 20:12:04.288342 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.288299    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-routing-overlay-volume" pod "03a937b1-fc8f-41ed-8cc8-ef54e20bae06" (UID: "03a937b1-fc8f-41ed-8cc8-ef54e20bae06"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:04.288590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.288562    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-plugins-config-volume" pod "03a937b1-fc8f-41ed-8cc8-ef54e20bae06" (UID: "03a937b1-fc8f-41ed-8cc8-ef54e20bae06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:04.288789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.288774    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:04.288836 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.288792    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:04.290237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.290221    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-provider-credentials" pod "03a937b1-fc8f-41ed-8cc8-ef54e20bae06" (UID: "03a937b1-fc8f-41ed-8cc8-ef54e20bae06"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:04.290320 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.290295    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-kube-api-access-v6cnr" pod "03a937b1-fc8f-41ed-8cc8-ef54e20bae06" (UID: "03a937b1-fc8f-41ed-8cc8-ef54e20bae06"). InnerVolumeSpecName "kube-api-access-v6cnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:04.390194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.390097    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:04.390194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.390133    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6cnr\" (UniqueName: \"kubernetes.io/projected/03a937b1-fc8f-41ed-8cc8-ef54e20bae06-kube-api-access-v6cnr\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:04.833655 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.833615    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:04.920784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.920751    5659 generic.go:356] "Generic (PLEG): container finished" podID="03a937b1-fc8f-41ed-8cc8-ef54e20bae06" containerID="0d8c6078b5dc6c23f9414f493bf8d985bb3fc61e2488f90b1bbf4ce3c5c44e69" exitCode=0
Nov 03 20:12:04.920980 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.920826    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"
Nov 03 20:12:04.920980 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.920834    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn" event={"ID":"03a937b1-fc8f-41ed-8cc8-ef54e20bae06","Type":"ContainerDied","Data":"0d8c6078b5dc6c23f9414f493bf8d985bb3fc61e2488f90b1bbf4ce3c5c44e69"}
Nov 03 20:12:04.920980 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.920893    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn" event={"ID":"03a937b1-fc8f-41ed-8cc8-ef54e20bae06","Type":"ContainerDied","Data":"7bb2e499c451624feb15195c6e78ec21e930325a16486a5e7eaaad7df79c09b8"}
Nov 03 20:12:04.920980 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.920915    5659 scope.go:122] "RemoveContainer" containerID="0d8c6078b5dc6c23f9414f493bf8d985bb3fc61e2488f90b1bbf4ce3c5c44e69"
Nov 03 20:12:04.950914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.950886    5659 scope.go:122] "RemoveContainer" containerID="0d8c6078b5dc6c23f9414f493bf8d985bb3fc61e2488f90b1bbf4ce3c5c44e69"
Nov 03 20:12:04.954942 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:12:04.954912    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8c6078b5dc6c23f9414f493bf8d985bb3fc61e2488f90b1bbf4ce3c5c44e69\": container with ID starting with 0d8c6078b5dc6c23f9414f493bf8d985bb3fc61e2488f90b1bbf4ce3c5c44e69 not found: ID does not exist" containerID="0d8c6078b5dc6c23f9414f493bf8d985bb3fc61e2488f90b1bbf4ce3c5c44e69"
Nov 03 20:12:04.955060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.954954    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8c6078b5dc6c23f9414f493bf8d985bb3fc61e2488f90b1bbf4ce3c5c44e69"} err="failed to get container status \"0d8c6078b5dc6c23f9414f493bf8d985bb3fc61e2488f90b1bbf4ce3c5c44e69\": rpc error: code = NotFound desc = could not find container \"0d8c6078b5dc6c23f9414f493bf8d985bb3fc61e2488f90b1bbf4ce3c5c44e69\": container with ID starting with 0d8c6078b5dc6c23f9414f493bf8d985bb3fc61e2488f90b1bbf4ce3c5c44e69 not found: ID does not exist"
Nov 03 20:12:04.956191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.956164    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"]
Nov 03 20:12:04.962607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:04.962582    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fcxm2rn"]
Nov 03 20:12:05.668666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:05.668636    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:05.974247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:05.974209    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a937b1-fc8f-41ed-8cc8-ef54e20bae06" path="/var/lib/kubelet/pods/03a937b1-fc8f-41ed-8cc8-ef54e20bae06/volumes"
Nov 03 20:12:06.323684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:06.323594    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:07.063834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:07.063799    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:07.747521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:07.747471    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:08.505028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:08.504984    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:09.136431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:09.136383    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:09.881872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:09.881836    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:10.688170 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:10.688130    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:11.436980 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:11.436948    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:12.278679 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:12.278644    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:13.150775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:13.150743    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:14.711371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:14.711335    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:15.226890 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:12:15.226844    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:12:15.689220 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:15.689107    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:16.338987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:16.338947    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:16.476785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:16.476735    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:16.985832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:16.985796    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:17.259989 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.259891    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:17.391303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.391270    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:17.559476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.559382    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"]
Nov 03 20:12:17.560196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.560177    5659 cpu_manager.go:414] "containerMap: removing container" podUID="03a937b1-fc8f-41ed-8cc8-ef54e20bae06" containerName="payload-processing"
Nov 03 20:12:17.560282 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.560197    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a937b1-fc8f-41ed-8cc8-ef54e20bae06" containerName="payload-processing"
Nov 03 20:12:17.560340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.560335    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a937b1-fc8f-41ed-8cc8-ef54e20bae06" containerName="payload-processing"
Nov 03 20:12:17.573123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.566133    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:17.577154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.577125    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"routing-overlay\""
Nov 03 20:12:17.578159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.578136    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"agc-cred-9d0b4d393cba\""
Nov 03 20:12:17.578378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.578363    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"agc-cred-2043ca6290ee\""
Nov 03 20:12:17.578560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.578545    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"agc-cred-f7b72c80fb02\""
Nov 03 20:12:17.578794 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.578775    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"kube-root-ca.crt\""
Nov 03 20:12:17.614545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.598804    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"]
Nov 03 20:12:17.697506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.674832    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-46a6d7\""
Nov 03 20:12:17.697506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.674954    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"openshift-service-ca.crt\""
Nov 03 20:12:17.697506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.675072    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"payload-processing-external-model-e2e--a1bcdb2a-dockercfg-5sh7w\""
Nov 03 20:12:17.733504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.699433    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2sxf\" (UniqueName: \"kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-kube-api-access-z2sxf\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:17.741506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.733923    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:17.765507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.754570    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:17.765507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.754909    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:17.930917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.906458    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:17.930917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.906620    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2sxf\" (UniqueName: \"kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-kube-api-access-z2sxf\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:17.949340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.932631    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:17.949340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.932684    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:17.949340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.934391    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:17.949340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.934918    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:17.949340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.936910    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:17.953043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:17.953003    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:18.051203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.051169    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2sxf\" (UniqueName: \"kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-kube-api-access-z2sxf\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:18.297508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.286734    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:18.348509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.323412    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:18.525511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.524986    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:18.638558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.638269    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"]
Nov 03 20:12:18.729698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.728582    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:18.761307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.759812    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"]
Nov 03 20:12:18.761307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.760145    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"agc-cred-de183148363e\""
Nov 03 20:12:18.895861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.817880    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"payload-processing-external-model-e2e--3d0fafa7-dockercfg-hbrjq\""
Nov 03 20:12:18.895861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.839894    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"kube-root-ca.crt\""
Nov 03 20:12:18.895861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.840044    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"openshift-service-ca.crt\""
Nov 03 20:12:18.895861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.889104    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8b7351\""
Nov 03 20:12:18.921533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.921034    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"routing-overlay\""
Nov 03 20:12:18.937806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.937764    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:18.937964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.937947    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:18.977516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.941645    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w864z\" (UniqueName: \"kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-kube-api-access-w864z\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:18.977691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:18.977521    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:19.090524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:19.090468    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:19.090861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:19.090841    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:19.091084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:19.091066    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w864z\" (UniqueName: \"kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-kube-api-access-w864z\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:19.091259 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:19.091243    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:19.092336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:19.092315    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:19.093700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:19.093068    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:19.152689 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:19.147083    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:19.173543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:19.173509    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w864z\" (UniqueName: \"kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-kube-api-access-w864z\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:19.244754 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:19.244718    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:19.367359 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:19.367312    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:19.472255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:19.472206    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"]
Nov 03 20:12:19.672396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:19.672368    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"]
Nov 03 20:12:19.673407 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:12:19.673385    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d39e4d_fbd5_4b91_82bb_4829b23137de.slice/crio-3b8abfb2308a7c56c39937e2708015ef0454da4d12bd69676aefe7ce9d4324ed WatchSource:0}: Error finding container 3b8abfb2308a7c56c39937e2708015ef0454da4d12bd69676aefe7ce9d4324ed: Status 404 returned error can't find the container with id 3b8abfb2308a7c56c39937e2708015ef0454da4d12bd69676aefe7ce9d4324ed
Nov 03 20:12:19.825932 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:19.825888    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:20.194554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:20.194515    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s" event={"ID":"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4","Type":"ContainerStarted","Data":"93e4d4b0463b88d77ebfb398fc1748f247281ea0055c25f7c13b07c7684d6069"}
Nov 03 20:12:20.194775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:20.194585    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s" event={"ID":"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4","Type":"ContainerStarted","Data":"97f8a9cf265bcead4be252ff2040772ae90b236947a9d9572c92963b9993e8e1"}
Nov 03 20:12:20.194775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:20.194672    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:20.202549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:20.202517    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn" event={"ID":"86d39e4d-fbd5-4b91-82bb-4829b23137de","Type":"ContainerStarted","Data":"14973618a0a8a08600408b85d60d4287e922b2d797715e3bcf267fd93d98ac32"}
Nov 03 20:12:20.202719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:20.202574    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:20.202719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:20.202593    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn" event={"ID":"86d39e4d-fbd5-4b91-82bb-4829b23137de","Type":"ContainerStarted","Data":"3b8abfb2308a7c56c39937e2708015ef0454da4d12bd69676aefe7ce9d4324ed"}
Nov 03 20:12:20.219034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:20.218978    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s" podStartSLOduration=3.218964944 podStartE2EDuration="3.218964944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:20.215409343 +0000 UTC m=+2092.370311762" watchObservedRunningTime="2026-11-03 20:12:20.218964944 +0000 UTC m=+2092.373867363"
Nov 03 20:12:20.238330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:20.238276    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:20.242659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:20.242604    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn" podStartSLOduration=2.242585276 podStartE2EDuration="2.242585276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:20.235052265 +0000 UTC m=+2092.389954684" watchObservedRunningTime="2026-11-03 20:12:20.242585276 +0000 UTC m=+2092.397487697"
Nov 03 20:12:20.946686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:20.946651    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:21.668116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:21.668080    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:22.457684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:22.457650    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:23.333229 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:23.333196    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:23.883383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:23.883350    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:24.026850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:24.026821    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:24.261022 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:24.259222    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:25.056505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:25.048564    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"]
Nov 03 20:12:25.193854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:25.193825    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:25.268098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:25.268065    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:25.401153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:25.401071    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:25.909853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:25.909822    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:26.122503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:26.058465    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:26.761997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:26.761940    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:27.602261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:27.602226    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:28.400103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:28.400075    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:29.127363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:29.111351    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:29.762197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:29.762156    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:30.049388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.049299    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn" podUID="86d39e4d-fbd5-4b91-82bb-4829b23137de" containerName="payload-processing" containerID="cri-o://14973618a0a8a08600408b85d60d4287e922b2d797715e3bcf267fd93d98ac32" gracePeriod=25
Nov 03 20:12:30.345365 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.345336    5659 generic.go:356] "Generic (PLEG): container finished" podID="86d39e4d-fbd5-4b91-82bb-4829b23137de" containerID="14973618a0a8a08600408b85d60d4287e922b2d797715e3bcf267fd93d98ac32" exitCode=0
Nov 03 20:12:30.345843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.345365    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn" event={"ID":"86d39e4d-fbd5-4b91-82bb-4829b23137de","Type":"ContainerDied","Data":"14973618a0a8a08600408b85d60d4287e922b2d797715e3bcf267fd93d98ac32"}
Nov 03 20:12:30.345843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.345430    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn" event={"ID":"86d39e4d-fbd5-4b91-82bb-4829b23137de","Type":"ContainerDied","Data":"3b8abfb2308a7c56c39937e2708015ef0454da4d12bd69676aefe7ce9d4324ed"}
Nov 03 20:12:30.345843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.345451    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b8abfb2308a7c56c39937e2708015ef0454da4d12bd69676aefe7ce9d4324ed"
Nov 03 20:12:30.346991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.346968    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:30.397866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.391826    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-routing-overlay-volume\") pod \"86d39e4d-fbd5-4b91-82bb-4829b23137de\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") "
Nov 03 20:12:30.397866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.391900    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-plugins-config-volume\") pod \"86d39e4d-fbd5-4b91-82bb-4829b23137de\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") "
Nov 03 20:12:30.397866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.392027    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-kube-api-access-w864z\" (UniqueName: \"kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-kube-api-access-w864z\") pod \"86d39e4d-fbd5-4b91-82bb-4829b23137de\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") "
Nov 03 20:12:30.397866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.392243    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-provider-credentials\" (UniqueName: \"kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-provider-credentials\") pod \"86d39e4d-fbd5-4b91-82bb-4829b23137de\" (UID: \"86d39e4d-fbd5-4b91-82bb-4829b23137de\") "
Nov 03 20:12:30.397866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.394056    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-routing-overlay-volume" pod "86d39e4d-fbd5-4b91-82bb-4829b23137de" (UID: "86d39e4d-fbd5-4b91-82bb-4829b23137de"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:30.397866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.394269    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-plugins-config-volume" pod "86d39e4d-fbd5-4b91-82bb-4829b23137de" (UID: "86d39e4d-fbd5-4b91-82bb-4829b23137de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:30.400082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.400042    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-provider-credentials" pod "86d39e4d-fbd5-4b91-82bb-4829b23137de" (UID: "86d39e4d-fbd5-4b91-82bb-4829b23137de"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:30.401034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.401006    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-kube-api-access-w864z" pod "86d39e4d-fbd5-4b91-82bb-4829b23137de" (UID: "86d39e4d-fbd5-4b91-82bb-4829b23137de"). InnerVolumeSpecName "kube-api-access-w864z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:30.494999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.494962    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:30.494999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.494996    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d39e4d-fbd5-4b91-82bb-4829b23137de-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:30.494999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.495005    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w864z\" (UniqueName: \"kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-kube-api-access-w864z\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:30.495235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.495014    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86d39e4d-fbd5-4b91-82bb-4829b23137de-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:30.583946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:30.583858    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:31.214384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:31.214353    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:12:31.269858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:31.269821    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:31.358518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:31.353631    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"
Nov 03 20:12:31.381842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:31.381815    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"]
Nov 03 20:12:31.411933 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:31.397835    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6b89tndvn"]
Nov 03 20:12:31.975497 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:31.975448    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d39e4d-fbd5-4b91-82bb-4829b23137de" path="/var/lib/kubelet/pods/86d39e4d-fbd5-4b91-82bb-4829b23137de/volumes"
Nov 03 20:12:32.031478 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:32.031446    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:32.753466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:32.753428    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:33.009177 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.009092    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"]
Nov 03 20:12:33.010302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.010271    5659 cpu_manager.go:414] "containerMap: removing container" podUID="86d39e4d-fbd5-4b91-82bb-4829b23137de" containerName="payload-processing"
Nov 03 20:12:33.010302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.010300    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d39e4d-fbd5-4b91-82bb-4829b23137de" containerName="payload-processing"
Nov 03 20:12:33.011635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.010508    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d39e4d-fbd5-4b91-82bb-4829b23137de" containerName="payload-processing"
Nov 03 20:12:33.065547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.063660    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.074577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.074542    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"]
Nov 03 20:12:33.113386 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.113353    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8b7351-openshift-default-dockercfg-lx6hb\""
Nov 03 20:12:33.168565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.168464    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.168769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.168628    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.168819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.168804    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.168952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.168934    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.169039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.169019    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.169101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.169084    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.169155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.169142    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.169293 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.169268    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d468b03c-81e8-44a3-948e-ed1e86de9aca-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.169387 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.169369    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l97t\" (UniqueName: \"kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-kube-api-access-8l97t\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.271423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.271340    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.271606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.271436    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.271606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.271527    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.271606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.271571    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.271788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.271620    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.271788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.271666    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d468b03c-81e8-44a3-948e-ed1e86de9aca-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.271788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.271746    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l97t\" (UniqueName: \"kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-kube-api-access-8l97t\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.271961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.271809    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.271961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.271857    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.271961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.271944    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.272188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.271982    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.272250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.272231    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.272429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.272393    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d468b03c-81e8-44a3-948e-ed1e86de9aca-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.272578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.272525    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.274323 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.274293    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.274572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.274550    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.284318 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.284286    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l97t\" (UniqueName: \"kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-kube-api-access-8l97t\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.284318 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.284296    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.394273 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.394229    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:33.448338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.448287    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:33.727081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.727050    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"]
Nov 03 20:12:33.727220 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:12:33.727165    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd468b03c_81e8_44a3_948e_ed1e86de9aca.slice/crio-05c45c422883690b31e604243a24d22b2efeeac63514417bc516be600adc4a2c WatchSource:0}: Error finding container 05c45c422883690b31e604243a24d22b2efeeac63514417bc516be600adc4a2c: Status 404 returned error can't find the container with id 05c45c422883690b31e604243a24d22b2efeeac63514417bc516be600adc4a2c
Nov 03 20:12:33.730069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.730030    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:33.730206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.730114    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:33.730206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:33.730187    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:34.240091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:34.240053    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:34.381119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:34.381076    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f" event={"ID":"d468b03c-81e8-44a3-948e-ed1e86de9aca","Type":"ContainerStarted","Data":"2b1977099f6cc32e2b75fcf97d7d3929e9bbed5fffe3c68bcbcf1db1bebc24a7"}
Nov 03 20:12:34.381343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:34.381144    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f" event={"ID":"d468b03c-81e8-44a3-948e-ed1e86de9aca","Type":"ContainerStarted","Data":"05c45c422883690b31e604243a24d22b2efeeac63514417bc516be600adc4a2c"}
Nov 03 20:12:34.395076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:34.395040    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:34.396964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:34.396941    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 20:12:34.397049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:34.396985    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f" podUID="d468b03c-81e8-44a3-948e-ed1e86de9aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 20:12:34.409355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:34.409309    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f" podStartSLOduration=2.40929683 podStartE2EDuration="2.40929683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:34.403825319 +0000 UTC m=+2106.558727738" watchObservedRunningTime="2026-11-03 20:12:34.40929683 +0000 UTC m=+2106.564199249"
Nov 03 20:12:34.971586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:34.971548    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:35.394949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:35.394862    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 20:12:35.394949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:35.394911    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f" podUID="d468b03c-81e8-44a3-948e-ed1e86de9aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 20:12:35.753507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:35.752943    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:36.396634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:36.396596    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 20:12:36.397103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:36.396657    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f" podUID="d468b03c-81e8-44a3-948e-ed1e86de9aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 20:12:36.682060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:36.681960    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:37.410505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:37.398962    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 20:12:37.410505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:37.399030    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f" podUID="d468b03c-81e8-44a3-948e-ed1e86de9aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 20:12:37.889231 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:37.889141    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:38.395155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:38.395115    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 20:12:38.395348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:38.395176    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f" podUID="d468b03c-81e8-44a3-948e-ed1e86de9aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 20:12:38.444068 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:38.443948    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:38.759507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:38.759436    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:39.065242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.065147    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"]
Nov 03 20:12:39.070496 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.070460    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.077619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.077594    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e4ce7-openshift-default-dockercfg-nppzh\""
Nov 03 20:12:39.081860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.081832    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"]
Nov 03 20:12:39.126110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.126060    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.126301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.126176    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-istio-token\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.126301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.126245    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-data\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.126413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.126328    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-certs\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.126413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.126401    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-credential-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.126530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.126477    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8mxx\" (UniqueName: \"kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-kube-api-access-v8mxx\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.126603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.126572    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-envoy\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.126666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.126605    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040df513-c1c5-424f-aa74-89c927dc28be-istio-podinfo\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.126666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.126631    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040df513-c1c5-424f-aa74-89c927dc28be-istiod-ca-cert\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.236519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.236454    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-istio-token\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.237230 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.237201    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-data\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.238120 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.238012    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-data\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.238385 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.238366    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-certs\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.238936 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.238830    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-certs\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.239128 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.239111    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-credential-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.239675 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.239505    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-credential-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.239901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.239881    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8mxx\" (UniqueName: \"kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-kube-api-access-v8mxx\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.240682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.240659    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-envoy\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.241424 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.241280    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040df513-c1c5-424f-aa74-89c927dc28be-istio-podinfo\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.242046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.242025    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040df513-c1c5-424f-aa74-89c927dc28be-istiod-ca-cert\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.243007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.242901    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040df513-c1c5-424f-aa74-89c927dc28be-istiod-ca-cert\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.243174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.243157    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.243762 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.243631    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.284533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.281385    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040df513-c1c5-424f-aa74-89c927dc28be-istio-podinfo\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.284533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.282099    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-envoy\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.284533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.284311    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-istio-token\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.286155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.286132    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8mxx\" (UniqueName: \"kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-kube-api-access-v8mxx\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.390546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.390444    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:39.394813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.394766    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 20:12:39.394970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.394857    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f" podUID="d468b03c-81e8-44a3-948e-ed1e86de9aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 20:12:39.749834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.730872    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:39.749834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.739678    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"]
Nov 03 20:12:39.833302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.832023    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:39.833302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.832119    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:39.833302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:39.832164    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:40.394951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:40.394821    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 20:12:40.394951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:40.394879    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f" podUID="d468b03c-81e8-44a3-948e-ed1e86de9aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 20:12:40.487710 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:40.487668    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84" event={"ID":"040df513-c1c5-424f-aa74-89c927dc28be","Type":"ContainerStarted","Data":"75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d"}
Nov 03 20:12:40.487891 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:40.487736    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84" event={"ID":"040df513-c1c5-424f-aa74-89c927dc28be","Type":"ContainerStarted","Data":"fcc4b799cbb55c71ce38cf67e1834b5e2b2ac31bae8b602ccab16e1ca5bfbbc0"}
Nov 03 20:12:40.517053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:40.516994    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84" podStartSLOduration=1.5169791209999999 podStartE2EDuration="1.516979121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:40.512677408 +0000 UTC m=+2112.667579830" watchObservedRunningTime="2026-11-03 20:12:40.516979121 +0000 UTC m=+2112.671881540"
Nov 03 20:12:40.608452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:40.608417    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:40.761549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:40.761443    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:41.305583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:41.305551    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:41.421508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:41.413005    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:41.421508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:41.417826    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:12:41.421508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:41.417930    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84" podUID="040df513-c1c5-424f-aa74-89c927dc28be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:12:41.466530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:41.465656    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:41.476518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:41.467445    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:41.502508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:41.500167    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:41.606607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:41.605822    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:12:42.274506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:42.273988    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:42.421001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:42.419131    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:12:42.421001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:42.419203    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84" podUID="040df513-c1c5-424f-aa74-89c927dc28be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:12:42.423295 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:42.423262    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:43.301413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.301371    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:43.398509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.393572    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:12:43.398509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.393646    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84" podUID="040df513-c1c5-424f-aa74-89c927dc28be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:12:43.828396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.828361    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"]
Nov 03 20:12:43.839778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.839749    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:43.886268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.886235    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"]
Nov 03 20:12:43.990299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.990266    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:43.990519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.990358    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:43.990519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.990401    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:43.990640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.990573    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f421d2e3-252b-45a6-b625-838fa53361c2-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:43.990640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.990607    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgnrs\" (UniqueName: \"kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-kube-api-access-wgnrs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:43.990759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.990697    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:43.990759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.990734    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:43.991551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.991521    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f421d2e3-252b-45a6-b625-838fa53361c2-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:43.991658 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:43.991643    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.093671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.093573    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f421d2e3-252b-45a6-b625-838fa53361c2-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.093860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.093671    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.094117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.094091    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.094228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.094162    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.094228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.094166    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.094340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.094229    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.094398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.094345    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f421d2e3-252b-45a6-b625-838fa53361c2-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.094398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.094375    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgnrs\" (UniqueName: \"kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-kube-api-access-wgnrs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.094559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.094444    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.094559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.094511    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.094682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.094560    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.094838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.094812    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.094907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.094859    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.095021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.095004    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f421d2e3-252b-45a6-b625-838fa53361c2-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.096344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.096322    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f421d2e3-252b-45a6-b625-838fa53361c2-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.096533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.096515    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.107229 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.107203    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.107732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.107714    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgnrs\" (UniqueName: \"kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-kube-api-access-wgnrs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.163432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.163344    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:44.194840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.194805    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:44.399395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.398874    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:12:44.399395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.398931    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84" podUID="040df513-c1c5-424f-aa74-89c927dc28be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:12:44.588007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.587960    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"]
Nov 03 20:12:44.599371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.599331    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:44.599462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.599423    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:44.599539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.599470    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:44.707101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:44.707057    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c" event={"ID":"f421d2e3-252b-45a6-b625-838fa53361c2","Type":"ContainerStarted","Data":"c847af84775615425e429b3566524fa2f326e58010efcc36411836ffe46550ac"}
Nov 03 20:12:45.085508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:45.084537    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:45.233513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:45.231788    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:45.282631 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:12:45.281800    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:12:45.381207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:45.378991    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:45.393856 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:45.393818    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:12:45.394054 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:45.393879    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84" podUID="040df513-c1c5-424f-aa74-89c927dc28be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:12:45.749648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:45.749596    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c" event={"ID":"f421d2e3-252b-45a6-b625-838fa53361c2","Type":"ContainerStarted","Data":"2b95e4f4a0f160d5565ba937a78dafb461bb5961ee8fdd8a0769311d6ec8df93"}
Nov 03 20:12:45.799526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:45.799181    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c" podStartSLOduration=2.799162888 podStartE2EDuration="2.799162888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:45.793436524 +0000 UTC m=+2117.948338943" watchObservedRunningTime="2026-11-03 20:12:45.799162888 +0000 UTC m=+2117.954065308"
Nov 03 20:12:46.204378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:46.199568    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:46.206398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:46.204898    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:12:46.206398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:46.204956    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c" podUID="f421d2e3-252b-45a6-b625-838fa53361c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:12:46.428949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:46.425828    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:12:46.428949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:46.425894    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84" podUID="040df513-c1c5-424f-aa74-89c927dc28be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:12:46.466232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:46.466146    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:46.645506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:46.625302    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:46.885743 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:46.885545    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:46.896299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:46.896266    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"]
Nov 03 20:12:46.909897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:46.909861    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:46.923242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:46.923199    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"]
Nov 03 20:12:47.105579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.105540    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:47.105848 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.105823    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:47.106083 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.106056    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:47.106247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.106228    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jw4j\" (UniqueName: \"kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-kube-api-access-5jw4j\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:47.200093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.200012    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:12:47.200305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.200284    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c" podUID="f421d2e3-252b-45a6-b625-838fa53361c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:12:47.208180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.208151    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:47.208586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.208555    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:47.208690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.208610    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jw4j\" (UniqueName: \"kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-kube-api-access-5jw4j\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:47.208846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.208819    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:47.208927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.208836    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:47.209448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.209418    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:47.212389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.212362    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:47.228060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.228035    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jw4j\" (UniqueName: \"kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-kube-api-access-5jw4j\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:47.306007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.305710    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:47.403521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.402816    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:47.403521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.403239    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:47.404308 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.404289    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:12:47.686321 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:47.686281    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"]
Nov 03 20:12:47.693826 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:12:47.692445    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c5e5cf_982c_4050_a373_b2ad41e6bae1.slice/crio-c046d7d1b2531719395bb079f73483b201278e551c6d39dd944e52e8350b9a5f WatchSource:0}: Error finding container c046d7d1b2531719395bb079f73483b201278e551c6d39dd944e52e8350b9a5f: Status 404 returned error can't find the container with id c046d7d1b2531719395bb079f73483b201278e551c6d39dd944e52e8350b9a5f
Nov 03 20:12:48.054567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:48.050111    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:48.078143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:48.078104    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz" event={"ID":"67c5e5cf-982c-4050-a373-b2ad41e6bae1","Type":"ContainerStarted","Data":"b474aebf6c87bf029a067fcca386995e496bfac8d7702a9bfbdb8cef25bfab36"}
Nov 03 20:12:48.078340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:48.078162    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz" event={"ID":"67c5e5cf-982c-4050-a373-b2ad41e6bae1","Type":"ContainerStarted","Data":"c046d7d1b2531719395bb079f73483b201278e551c6d39dd944e52e8350b9a5f"}
Nov 03 20:12:48.079279 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:48.079254    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:12:48.114581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:48.114520    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz" podStartSLOduration=2.114502168 podStartE2EDuration="2.114502168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:48.113048704 +0000 UTC m=+2120.267951125" watchObservedRunningTime="2026-11-03 20:12:48.114502168 +0000 UTC m=+2120.269404585"
Nov 03 20:12:48.192508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:48.192451    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:48.199705 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:48.199674    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:12:48.199914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:48.199728    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c" podUID="f421d2e3-252b-45a6-b625-838fa53361c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:12:48.687011 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:48.686957    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:49.200159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:49.200122    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:12:49.200696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:49.200183    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c" podUID="f421d2e3-252b-45a6-b625-838fa53361c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:12:49.549640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:49.549606    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:50.200468 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:50.200435    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:12:50.200468 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:50.200501    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c" podUID="f421d2e3-252b-45a6-b625-838fa53361c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:12:50.257995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:50.257958    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:50.970406 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:50.970369    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:51.200333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:51.200298    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 03 20:12:51.200503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:51.200352    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c" podUID="f421d2e3-252b-45a6-b625-838fa53361c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 03 20:12:51.637214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:51.637173    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:52.230073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:52.230038    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:52.230561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:52.230534    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:52.231597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:52.231577    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:12:52.358101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:52.358070    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:53.093748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:53.093715    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:53.846292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:53.846259    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:53.971034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:53.971000    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:54.103952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:54.103850    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:54.577733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:54.577703    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:54.716001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:54.715947    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:54.845143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:54.845052    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:54.981509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:54.975124    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:55.636230 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:55.636194    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:56.357337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:56.357302    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:57.165843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:57.163703    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:57.824866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:57.824835    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:58.515074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:58.515040    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:59.264911 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:59.264871    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:59.988536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:12:59.972775    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:00.098609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:00.098361    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:00.100505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:00.100078    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:13:00.180341 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:00.180294    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"]
Nov 03 20:13:00.232183 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:00.232148    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-46a6d7/agc-cred-2043ca6290ee: secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:00.232459 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:00.232440    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz: secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:00.232682 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:00.232667    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials podName:67c5e5cf-982c-4050-a373-b2ad41e6bae1 nodeName:}" failed. No retries permitted until 2026-11-03 20:13:00.732644367 +0000 UTC m=+2132.887546784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz" (UID: "67c5e5cf-982c-4050-a373-b2ad41e6bae1") : secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:00.244392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:00.244312    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:00.812687 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:00.812635    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-46a6d7/agc-cred-2043ca6290ee: secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:00.812687 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:00.812669    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz: secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:00.812964 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:00.812743    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials podName:67c5e5cf-982c-4050-a373-b2ad41e6bae1 nodeName:}" failed. No retries permitted until 2026-11-03 20:13:01.812729525 +0000 UTC m=+2133.967631935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz" (UID: "67c5e5cf-982c-4050-a373-b2ad41e6bae1") : secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:01.306756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:01.306720    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:01.834394 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:01.834355    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-46a6d7/agc-cred-2043ca6290ee: secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:01.834394 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:01.834390    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz: secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:01.834706 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:01.834446    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials podName:67c5e5cf-982c-4050-a373-b2ad41e6bae1 nodeName:}" failed. No retries permitted until 2026-11-03 20:13:03.834431291 +0000 UTC m=+2135.989333687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz" (UID: "67c5e5cf-982c-4050-a373-b2ad41e6bae1") : secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:01.933907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:01.933870    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:02.066226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:02.066189    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:02.195832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:02.195742    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:02.351574 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:02.348982    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:03.147121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:03.147090    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:03.832563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:03.832527    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:03.881847 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:03.881817    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-46a6d7/agc-cred-2043ca6290ee: secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:03.881847 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:03.881847    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz: secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:03.882097 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:03.881917    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials podName:67c5e5cf-982c-4050-a373-b2ad41e6bae1 nodeName:}" failed. No retries permitted until 2026-11-03 20:13:07.881901305 +0000 UTC m=+2140.036803716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz" (UID: "67c5e5cf-982c-4050-a373-b2ad41e6bae1") : secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:04.615505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:04.614805    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:05.181671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.181622    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s" podUID="48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4" containerName="payload-processing" containerID="cri-o://93e4d4b0463b88d77ebfb398fc1748f247281ea0055c25f7c13b07c7684d6069" gracePeriod=25
Nov 03 20:13:05.443571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.443400    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:05.556942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.556912    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:13:05.713264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.713227    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-plugins-config-volume\") pod \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") "
Nov 03 20:13:05.713451 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.713306    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-routing-overlay-volume\") pod \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") "
Nov 03 20:13:05.713524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.713459    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-kube-api-access-z2sxf\" (UniqueName: \"kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-kube-api-access-z2sxf\") pod \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") "
Nov 03 20:13:05.713571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.713527    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-provider-credentials\") pod \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\" (UID: \"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4\") "
Nov 03 20:13:05.713751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.713725    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-plugins-config-volume" pod "48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4" (UID: "48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:05.713806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.713732    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-routing-overlay-volume" pod "48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4" (UID: "48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:05.714434 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.714412    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:05.714565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.714441    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:05.715906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.715875    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-kube-api-access-z2sxf" pod "48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4" (UID: "48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4"). InnerVolumeSpecName "kube-api-access-z2sxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:05.716017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.715975    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-provider-credentials" pod "48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4" (UID: "48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:05.815706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.815667    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2sxf\" (UniqueName: \"kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-kube-api-access-z2sxf\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:05.815706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:05.815699    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:06.137720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:06.137681    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:06.282645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:06.282551    5659 generic.go:356] "Generic (PLEG): container finished" podID="48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4" containerID="93e4d4b0463b88d77ebfb398fc1748f247281ea0055c25f7c13b07c7684d6069" exitCode=0
Nov 03 20:13:06.282645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:06.282601    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s" event={"ID":"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4","Type":"ContainerDied","Data":"93e4d4b0463b88d77ebfb398fc1748f247281ea0055c25f7c13b07c7684d6069"}
Nov 03 20:13:06.282645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:06.282634    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"
Nov 03 20:13:06.283202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:06.282672    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s" event={"ID":"48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4","Type":"ContainerDied","Data":"97f8a9cf265bcead4be252ff2040772ae90b236947a9d9572c92963b9993e8e1"}
Nov 03 20:13:06.283202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:06.282705    5659 scope.go:122] "RemoveContainer" containerID="93e4d4b0463b88d77ebfb398fc1748f247281ea0055c25f7c13b07c7684d6069"
Nov 03 20:13:06.296237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:06.296211    5659 scope.go:122] "RemoveContainer" containerID="93e4d4b0463b88d77ebfb398fc1748f247281ea0055c25f7c13b07c7684d6069"
Nov 03 20:13:06.296609 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:06.296582    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e4d4b0463b88d77ebfb398fc1748f247281ea0055c25f7c13b07c7684d6069\": container with ID starting with 93e4d4b0463b88d77ebfb398fc1748f247281ea0055c25f7c13b07c7684d6069 not found: ID does not exist" containerID="93e4d4b0463b88d77ebfb398fc1748f247281ea0055c25f7c13b07c7684d6069"
Nov 03 20:13:06.296723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:06.296616    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e4d4b0463b88d77ebfb398fc1748f247281ea0055c25f7c13b07c7684d6069"} err="failed to get container status \"93e4d4b0463b88d77ebfb398fc1748f247281ea0055c25f7c13b07c7684d6069\": rpc error: code = NotFound desc = could not find container \"93e4d4b0463b88d77ebfb398fc1748f247281ea0055c25f7c13b07c7684d6069\": container with ID starting with 93e4d4b0463b88d77ebfb398fc1748f247281ea0055c25f7c13b07c7684d6069 not found: ID does not exist"
Nov 03 20:13:06.305609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:06.305580    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"]
Nov 03 20:13:06.309694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:06.309672    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-78fdnpf5s"]
Nov 03 20:13:06.843103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:06.843068    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:07.824823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:07.824790    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:07.976145 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:07.975039    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4" path="/var/lib/kubelet/pods/48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4/volumes"
Nov 03 20:13:07.977173 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:07.977135    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-46a6d7/agc-cred-2043ca6290ee: secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:07.977173 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:07.977178    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz: secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:07.977377 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:07.977278    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials podName:67c5e5cf-982c-4050-a373-b2ad41e6bae1 nodeName:}" failed. No retries permitted until 2026-11-03 20:13:15.977250171 +0000 UTC m=+2148.132152582 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz" (UID: "67c5e5cf-982c-4050-a373-b2ad41e6bae1") : secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:08.521810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:08.521779    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:08.657974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:08.657940    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:08.806708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:08.805382    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:09.073855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:09.072807    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:09.295863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:09.295828    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:09.450974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:09.450887    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:09.641859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:09.641824    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:09.779729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:09.779693    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:10.724104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:10.724065    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:10.871133 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:10.871099    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:11.005662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:11.005579    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:11.151250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:11.151217    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:11.291253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:11.291172    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:11.461332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:11.461298    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:11.611107 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:11.611024    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:11.756058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:11.756005    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:12.166902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:12.166845    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:12.298812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:12.298758    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:12.754655 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:12.754614    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"]
Nov 03 20:13:12.967986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:12.967952    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:13.631253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:13.631222    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:14.464132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:14.464097    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:15.151817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:15.151781    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:15.236163 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:15.236127    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:13:15.835350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:15.835309    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:16.035637 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:16.035137    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-46a6d7/agc-cred-2043ca6290ee: secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:16.035637 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:16.035170    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz: secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:16.035637 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:16.035237    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials podName:67c5e5cf-982c-4050-a373-b2ad41e6bae1 nodeName:}" failed. No retries permitted until 2026-11-03 20:13:32.0352158 +0000 UTC m=+2164.190118218 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz" (UID: "67c5e5cf-982c-4050-a373-b2ad41e6bae1") : secret "agc-cred-2043ca6290ee" not found
Nov 03 20:13:16.558507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:16.558454    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:16.734283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:16.734245    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:16.793509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:16.777275    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"]
Nov 03 20:13:16.793509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:16.778581    5659 cpu_manager.go:414] "containerMap: removing container" podUID="48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4" containerName="payload-processing"
Nov 03 20:13:16.793509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:16.778657    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4" containerName="payload-processing"
Nov 03 20:13:16.793509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:16.778939    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bf62cd-ff23-4cdc-88e5-1ce7a670e0d4" containerName="payload-processing"
Nov 03 20:13:16.807828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:16.807123    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:16.887506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:16.877105    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"]
Nov 03 20:13:16.907511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:16.889958    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfkfg\" (UniqueName: \"kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-kube-api-access-gfkfg\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:16.907902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:16.907871    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:16.914531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:16.908109    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:16.928503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:16.914865    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:17.031520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:17.030555    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:17.031520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:17.030650    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:17.031520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:17.030704    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:17.031839 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:17.031754    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:17.038509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:17.032266    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfkfg\" (UniqueName: \"kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-kube-api-access-gfkfg\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:17.038509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:17.032721    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:17.058278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:17.058196    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfkfg\" (UniqueName: \"kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-kube-api-access-gfkfg\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:17.060579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:17.060515    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:17.311136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:17.310816    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:17.757635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:17.757586    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz" podUID="67c5e5cf-982c-4050-a373-b2ad41e6bae1" containerName="payload-processing" containerID="cri-o://b474aebf6c87bf029a067fcca386995e496bfac8d7702a9bfbdb8cef25bfab36" gracePeriod=25
Nov 03 20:13:17.873115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:17.873080    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"]
Nov 03 20:13:17.874406 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:13:17.874371    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9b00f9_5c52_4a44_8f8e_96a5f8408118.slice/crio-30d647c139e1599885857549963ae2fec620cd83aba1d96accbf6b477e3ac733 WatchSource:0}: Error finding container 30d647c139e1599885857549963ae2fec620cd83aba1d96accbf6b477e3ac733: Status 404 returned error can't find the container with id 30d647c139e1599885857549963ae2fec620cd83aba1d96accbf6b477e3ac733
Nov 03 20:13:18.012072 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.011258    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:18.389007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.388967    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"]
Nov 03 20:13:18.389427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.389402    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz" podUID="a6671998-ae34-4319-8e68-9785a7d6558d" containerName="istio-proxy" containerID="cri-o://dd6324defaae3f0eceddaca8259fba2129d9868bf099d10ee180296fb93b5e1a" gracePeriod=30
Nov 03 20:13:18.390147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.390115    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:18.407661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.407638    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:13:18.455504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.453834    5659 generic.go:356] "Generic (PLEG): container finished" podID="67c5e5cf-982c-4050-a373-b2ad41e6bae1" containerID="b474aebf6c87bf029a067fcca386995e496bfac8d7702a9bfbdb8cef25bfab36" exitCode=0
Nov 03 20:13:18.455504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.454057    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz" event={"ID":"67c5e5cf-982c-4050-a373-b2ad41e6bae1","Type":"ContainerDied","Data":"b474aebf6c87bf029a067fcca386995e496bfac8d7702a9bfbdb8cef25bfab36"}
Nov 03 20:13:18.455504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.454107    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz" event={"ID":"67c5e5cf-982c-4050-a373-b2ad41e6bae1","Type":"ContainerDied","Data":"c046d7d1b2531719395bb079f73483b201278e551c6d39dd944e52e8350b9a5f"}
Nov 03 20:13:18.455504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.454136    5659 scope.go:122] "RemoveContainer" containerID="b474aebf6c87bf029a067fcca386995e496bfac8d7702a9bfbdb8cef25bfab36"
Nov 03 20:13:18.455504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.454275    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"
Nov 03 20:13:18.457601 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.457555    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg" event={"ID":"bc9b00f9-5c52-4a44-8f8e-96a5f8408118","Type":"ContainerStarted","Data":"9ee2d13554e2b96203d31137ff42dfce6ff2ab9a324e854c7608846ae9c3d1b4"}
Nov 03 20:13:18.457716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.457631    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg" event={"ID":"bc9b00f9-5c52-4a44-8f8e-96a5f8408118","Type":"ContainerStarted","Data":"30d647c139e1599885857549963ae2fec620cd83aba1d96accbf6b477e3ac733"}
Nov 03 20:13:18.458686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.458660    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:18.477331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.473081    5659 scope.go:122] "RemoveContainer" containerID="b474aebf6c87bf029a067fcca386995e496bfac8d7702a9bfbdb8cef25bfab36"
Nov 03 20:13:18.485528 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:18.485468    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b474aebf6c87bf029a067fcca386995e496bfac8d7702a9bfbdb8cef25bfab36\": container with ID starting with b474aebf6c87bf029a067fcca386995e496bfac8d7702a9bfbdb8cef25bfab36 not found: ID does not exist" containerID="b474aebf6c87bf029a067fcca386995e496bfac8d7702a9bfbdb8cef25bfab36"
Nov 03 20:13:18.485759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.485723    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b474aebf6c87bf029a067fcca386995e496bfac8d7702a9bfbdb8cef25bfab36"} err="failed to get container status \"b474aebf6c87bf029a067fcca386995e496bfac8d7702a9bfbdb8cef25bfab36\": rpc error: code = NotFound desc = could not find container \"b474aebf6c87bf029a067fcca386995e496bfac8d7702a9bfbdb8cef25bfab36\": container with ID starting with b474aebf6c87bf029a067fcca386995e496bfac8d7702a9bfbdb8cef25bfab36 not found: ID does not exist"
Nov 03 20:13:18.501449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.501394    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg" podStartSLOduration=2.501380086 podStartE2EDuration="2.501380086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:18.485726492 +0000 UTC m=+2150.640628912" watchObservedRunningTime="2026-11-03 20:13:18.501380086 +0000 UTC m=+2150.656282506"
Nov 03 20:13:18.555399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.555364    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-plugins-config-volume\") pod \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") "
Nov 03 20:13:18.555617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.555586    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-routing-overlay-volume\") pod \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") "
Nov 03 20:13:18.555700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.555645    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials\") pod \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") "
Nov 03 20:13:18.555700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.555677    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-kube-api-access-5jw4j\" (UniqueName: \"kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-kube-api-access-5jw4j\") pod \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\" (UID: \"67c5e5cf-982c-4050-a373-b2ad41e6bae1\") "
Nov 03 20:13:18.555850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.555822    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-plugins-config-volume" pod "67c5e5cf-982c-4050-a373-b2ad41e6bae1" (UID: "67c5e5cf-982c-4050-a373-b2ad41e6bae1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:18.556048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.556027    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-routing-overlay-volume" pod "67c5e5cf-982c-4050-a373-b2ad41e6bae1" (UID: "67c5e5cf-982c-4050-a373-b2ad41e6bae1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:18.556801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.556775    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:18.556902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.556806    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c5e5cf-982c-4050-a373-b2ad41e6bae1-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:18.558579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.558552    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-kube-api-access-5jw4j" pod "67c5e5cf-982c-4050-a373-b2ad41e6bae1" (UID: "67c5e5cf-982c-4050-a373-b2ad41e6bae1"). InnerVolumeSpecName "kube-api-access-5jw4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:18.558866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.558846    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials" pod "67c5e5cf-982c-4050-a373-b2ad41e6bae1" (UID: "67c5e5cf-982c-4050-a373-b2ad41e6bae1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:18.658368 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.658278    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:18.658368 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.658319    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jw4j\" (UniqueName: \"kubernetes.io/projected/67c5e5cf-982c-4050-a373-b2ad41e6bae1-kube-api-access-5jw4j\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:18.720776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.720722    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:18.801383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.801332    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"]
Nov 03 20:13:18.806107 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.806072    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495xrbjz"]
Nov 03 20:13:18.895000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:18.894955    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:19.136578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:19.135350    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:19.620550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:19.620515    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:19.813393 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:19.804372    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:19.974581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:19.974543    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c5e5cf-982c-4050-a373-b2ad41e6bae1" path="/var/lib/kubelet/pods/67c5e5cf-982c-4050-a373-b2ad41e6bae1/volumes"
Nov 03 20:13:20.136379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:20.136347    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:20.313669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:20.311412    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:20.906838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:20.906805    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:21.062717 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.062681    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:21.071237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.071204    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"]
Nov 03 20:13:21.072411 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.072386    5659 cpu_manager.go:414] "containerMap: removing container" podUID="67c5e5cf-982c-4050-a373-b2ad41e6bae1" containerName="payload-processing"
Nov 03 20:13:21.072411 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.072412    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c5e5cf-982c-4050-a373-b2ad41e6bae1" containerName="payload-processing"
Nov 03 20:13:21.072651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.072616    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c5e5cf-982c-4050-a373-b2ad41e6bae1" containerName="payload-processing"
Nov 03 20:13:21.078441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.078417    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.085548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.085525    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"openshift-service-ca.crt\""
Nov 03 20:13:21.087358 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.087338    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-98529d\""
Nov 03 20:13:21.087471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.087386    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"agc-cred-c4c7ed60cafb\""
Nov 03 20:13:21.087471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.087407    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"routing-overlay\""
Nov 03 20:13:21.087471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.087457    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"kube-root-ca.crt\""
Nov 03 20:13:21.087710 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.087591    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"payload-processing-external-model-e2e--c4b4e61e-dockercfg-8sdl4\""
Nov 03 20:13:21.092831 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.092806    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"]
Nov 03 20:13:21.172293 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.172191    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.172293 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.172243    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qphj\" (UniqueName: \"kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-kube-api-access-2qphj\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.172568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.172546    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.172749 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.172720    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.232406 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.232366    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:21.280634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.276733    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.280634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.276904    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.280634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.277052    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.280634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.277083    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qphj\" (UniqueName: \"kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-kube-api-access-2qphj\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.280634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.279378    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.299307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.294338    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.326510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.318114    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.374507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.369838    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qphj\" (UniqueName: \"kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-kube-api-access-2qphj\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.420523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.401939    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:21.442615 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.442535    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:21.831532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.828904    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"]
Nov 03 20:13:21.836954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.836823    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:21.846161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.846130    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"]
Nov 03 20:13:21.905511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.898287    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"payload-processing-external-model-e2e--088821c0-dockercfg-mqh9q\""
Nov 03 20:13:21.905511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.898420    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"agc-cred-f5fb49f604b6\""
Nov 03 20:13:21.905511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.898542    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4c2a28\""
Nov 03 20:13:21.905511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.898606    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"routing-overlay\""
Nov 03 20:13:21.905511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.898694    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"kube-root-ca.crt\""
Nov 03 20:13:21.905511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:21.898767    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"openshift-service-ca.crt\""
Nov 03 20:13:22.063506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.038256    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:22.063506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.038353    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:22.063506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.038399    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:22.063506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.038659    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrqx9\" (UniqueName: \"kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-kube-api-access-zrqx9\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:22.144832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.141152    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrqx9\" (UniqueName: \"kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-kube-api-access-zrqx9\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:22.144832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.141388    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:22.144832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.141465    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:22.144832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.141559    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:22.144832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.142982    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:22.145246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.144843    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:22.152654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.150832    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:22.158430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.158401    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrqx9\" (UniqueName: \"kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-kube-api-access-zrqx9\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:22.357706 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:13:22.355942    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600d6fa3_b642_454e_b90a_3f5499770f4b.slice/crio-1f7062bb34a8f777b5fe896d0cd83273765d0f198606e0264409946493cd48be WatchSource:0}: Error finding container 1f7062bb34a8f777b5fe896d0cd83273765d0f198606e0264409946493cd48be: Status 404 returned error can't find the container with id 1f7062bb34a8f777b5fe896d0cd83273765d0f198606e0264409946493cd48be
Nov 03 20:13:22.376292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.376177    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"]
Nov 03 20:13:22.402245 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.401960    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:22.520000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.519827    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:22.651842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.650978    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:22.771317 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.771289    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"]
Nov 03 20:13:22.800203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.800171    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:22.926610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.922790    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq" event={"ID":"600d6fa3-b642-454e-b90a-3f5499770f4b","Type":"ContainerStarted","Data":"942fcc710d5a2905d4afb412ddff6accaabb6b6e9f714d7eafb6d91e72aaeaf5"}
Nov 03 20:13:22.926610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.922857    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq" event={"ID":"600d6fa3-b642-454e-b90a-3f5499770f4b","Type":"ContainerStarted","Data":"1f7062bb34a8f777b5fe896d0cd83273765d0f198606e0264409946493cd48be"}
Nov 03 20:13:22.926610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:22.924137    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:23.020683 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.015509    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:23.035666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.035424    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982" event={"ID":"12961900-21e3-4340-8942-964f5588ae12","Type":"ContainerStarted","Data":"f207a89b0fa633e14224792bb1dd227709d4c28fb056b76680a2508a776ab3f9"}
Nov 03 20:13:23.123896 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:23.123282    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-98529d/agc-cred-c4c7ed60cafb: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:23.152700 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:23.152672    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:23.153062 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:23.153043    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials podName:600d6fa3-b642-454e-b90a-3f5499770f4b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:23.653014482 +0000 UTC m=+2155.807916896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq" (UID: "600d6fa3-b642-454e-b90a-3f5499770f4b") : secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:23.199608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.198078    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq" podStartSLOduration=2.198053812 podStartE2EDuration="2.198053812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:22.993858237 +0000 UTC m=+2155.148760660" watchObservedRunningTime="2026-11-03 20:13:23.198053812 +0000 UTC m=+2155.352956269"
Nov 03 20:13:23.262215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.259294    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"]
Nov 03 20:13:23.373067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.372997    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"]
Nov 03 20:13:23.373600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.373130    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.391207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.391178    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"e2e-cfl-8de789-provider-cred\""
Nov 03 20:13:23.497613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.497572    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.497769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.497621    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svls\" (UniqueName: \"kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-kube-api-access-7svls\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.497769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.497692    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.497863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.497783    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.575035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.574666    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:23.599735 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.599515    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.599920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.599817    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.599920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.599862    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svls\" (UniqueName: \"kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-kube-api-access-7svls\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.600060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.599975    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.611635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.601078    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.611635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.601770    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.613465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.613328    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.621052 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.621025    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svls\" (UniqueName: \"kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-kube-api-access-7svls\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.688913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.688879    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:13:23.702048 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:23.702005    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-98529d/agc-cred-c4c7ed60cafb: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:23.702048 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:23.702037    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:23.702304 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:23.702112    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials podName:600d6fa3-b642-454e-b90a-3f5499770f4b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:24.702086168 +0000 UTC m=+2156.856988580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq" (UID: "600d6fa3-b642-454e-b90a-3f5499770f4b") : secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:23.708255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.708223    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:23.720227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.720150    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:23.805799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.804612    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6671998-ae34-4319-8e68-9785a7d6558d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6671998-ae34-4319-8e68-9785a7d6558d-istiod-ca-cert\") pod \"a6671998-ae34-4319-8e68-9785a7d6558d\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") "
Nov 03 20:13:23.805799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.804671    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-data\") pod \"a6671998-ae34-4319-8e68-9785a7d6558d\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") "
Nov 03 20:13:23.805799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.804812    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-socket\") pod \"a6671998-ae34-4319-8e68-9785a7d6558d\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") "
Nov 03 20:13:23.805799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.804872    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-istio-token\" (UniqueName: \"kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-istio-token\") pod \"a6671998-ae34-4319-8e68-9785a7d6558d\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") "
Nov 03 20:13:23.805799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.805043    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6671998-ae34-4319-8e68-9785a7d6558d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6671998-ae34-4319-8e68-9785a7d6558d-istio-podinfo\") pod \"a6671998-ae34-4319-8e68-9785a7d6558d\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") "
Nov 03 20:13:23.805799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.805194    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-envoy\") pod \"a6671998-ae34-4319-8e68-9785a7d6558d\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") "
Nov 03 20:13:23.805799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.805229    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-kube-api-access-446tm\" (UniqueName: \"kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-kube-api-access-446tm\") pod \"a6671998-ae34-4319-8e68-9785a7d6558d\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") "
Nov 03 20:13:23.805799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.805280    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-certs\") pod \"a6671998-ae34-4319-8e68-9785a7d6558d\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") "
Nov 03 20:13:23.805799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.805318    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-credential-socket\") pod \"a6671998-ae34-4319-8e68-9785a7d6558d\" (UID: \"a6671998-ae34-4319-8e68-9785a7d6558d\") "
Nov 03 20:13:23.806756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.806527    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-credential-socket" pod "a6671998-ae34-4319-8e68-9785a7d6558d" (UID: "a6671998-ae34-4319-8e68-9785a7d6558d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:23.807132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.806927    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-data" pod "a6671998-ae34-4319-8e68-9785a7d6558d" (UID: "a6671998-ae34-4319-8e68-9785a7d6558d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:23.807704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.807388    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6671998-ae34-4319-8e68-9785a7d6558d-istiod-ca-cert" pod "a6671998-ae34-4319-8e68-9785a7d6558d" (UID: "a6671998-ae34-4319-8e68-9785a7d6558d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:23.808556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.808106    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-certs" pod "a6671998-ae34-4319-8e68-9785a7d6558d" (UID: "a6671998-ae34-4319-8e68-9785a7d6558d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:23.808556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.808286    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-socket" pod "a6671998-ae34-4319-8e68-9785a7d6558d" (UID: "a6671998-ae34-4319-8e68-9785a7d6558d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:23.810596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.810527    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6671998-ae34-4319-8e68-9785a7d6558d-istio-podinfo" pod "a6671998-ae34-4319-8e68-9785a7d6558d" (UID: "a6671998-ae34-4319-8e68-9785a7d6558d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:13:23.810779 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.810751    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-envoy" pod "a6671998-ae34-4319-8e68-9785a7d6558d" (UID: "a6671998-ae34-4319-8e68-9785a7d6558d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:23.811735 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.811698    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-kube-api-access-446tm" pod "a6671998-ae34-4319-8e68-9785a7d6558d" (UID: "a6671998-ae34-4319-8e68-9785a7d6558d"). InnerVolumeSpecName "kube-api-access-446tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:23.813055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.813038    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-istio-token" pod "a6671998-ae34-4319-8e68-9785a7d6558d" (UID: "a6671998-ae34-4319-8e68-9785a7d6558d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:23.897346 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.897308    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:23.909131 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.909094    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6671998-ae34-4319-8e68-9785a7d6558d-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:23.909131 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.909134    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:23.909360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.909151    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-446tm\" (UniqueName: \"kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-kube-api-access-446tm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:23.909360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.909167    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:23.909360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.909182    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:23.909360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.909196    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6671998-ae34-4319-8e68-9785a7d6558d-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:23.909360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.909209    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:23.909360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.909223    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6671998-ae34-4319-8e68-9785a7d6558d-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:23.909360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.909237    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6671998-ae34-4319-8e68-9785a7d6558d-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:23.913572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.913515    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"]
Nov 03 20:13:23.914769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.914742    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a6671998-ae34-4319-8e68-9785a7d6558d" containerName="istio-proxy"
Nov 03 20:13:23.914873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.914771    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6671998-ae34-4319-8e68-9785a7d6558d" containerName="istio-proxy"
Nov 03 20:13:23.915040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.915023    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6671998-ae34-4319-8e68-9785a7d6558d" containerName="istio-proxy"
Nov 03 20:13:23.930792 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.930754    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:23.945732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.944078    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"kube-root-ca.crt\""
Nov 03 20:13:23.945732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.944834    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"payload-processing-external-model-e2e--3d0fafa7-dockercfg-jbrch\""
Nov 03 20:13:23.945732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.944901    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"agc-cred-5506390946ca\""
Nov 03 20:13:23.945732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.944986    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8b7351\""
Nov 03 20:13:23.945732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.945111    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"routing-overlay\""
Nov 03 20:13:23.979736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:23.979644    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"]
Nov 03 20:13:24.011179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.011142    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:24.011365 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.011249    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:24.011461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.011367    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78g98\" (UniqueName: \"kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-kube-api-access-78g98\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:24.011838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.011808    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:24.064535 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.048767    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"openshift-service-ca.crt\""
Nov 03 20:13:24.147516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.143569    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:24.147516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.145681    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:24.147516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.145745    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:24.147516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.145840    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:24.147516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.145896    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78g98\" (UniqueName: \"kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-kube-api-access-78g98\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:24.147516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.146668    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:24.160509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.156370    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:24.169504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.165716    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"]
Nov 03 20:13:24.213334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.210331    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78g98\" (UniqueName: \"kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-kube-api-access-78g98\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:24.219572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.215763    5659 generic.go:356] "Generic (PLEG): container finished" podID="a6671998-ae34-4319-8e68-9785a7d6558d" containerID="dd6324defaae3f0eceddaca8259fba2129d9868bf099d10ee180296fb93b5e1a" exitCode=0
Nov 03 20:13:24.219572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.215868    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz" event={"ID":"a6671998-ae34-4319-8e68-9785a7d6558d","Type":"ContainerDied","Data":"dd6324defaae3f0eceddaca8259fba2129d9868bf099d10ee180296fb93b5e1a"}
Nov 03 20:13:24.219572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.215920    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz" event={"ID":"a6671998-ae34-4319-8e68-9785a7d6558d","Type":"ContainerDied","Data":"997b1618835a33607675887fc328d3443cd572120e8d573f8ed5f7d6b4704ab1"}
Nov 03 20:13:24.219572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.215951    5659 scope.go:122] "RemoveContainer" containerID="dd6324defaae3f0eceddaca8259fba2129d9868bf099d10ee180296fb93b5e1a"
Nov 03 20:13:24.219572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.216154    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"
Nov 03 20:13:24.225504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.222972    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:24.280885 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.280816    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"]
Nov 03 20:13:24.293973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.282904    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:24.447503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.441992    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982" event={"ID":"12961900-21e3-4340-8942-964f5588ae12","Type":"ContainerStarted","Data":"dc7879ad6793f5d42c4efdc1b999e817954ffb844f12c8c21026b8a46c820c18"}
Nov 03 20:13:24.447503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.442051    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"]
Nov 03 20:13:24.447503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.442087    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:24.447503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.442188    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.458380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.454961    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"kube-root-ca.crt\""
Nov 03 20:13:24.458380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.455839    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-e2e--b1878920-dockercfg-6sqzm\""
Nov 03 20:13:24.458380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.455987    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"agc-cred-6e3d6ce11b17\""
Nov 03 20:13:24.458380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.456144    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"agc-cred-1bd75abb0adc\""
Nov 03 20:13:24.458380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.456240    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"agc-cred-4cbea2d0a3c6\""
Nov 03 20:13:24.458380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.456512    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"routing-overlay\""
Nov 03 20:13:24.458380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.456587    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"openshift-service-ca.crt\""
Nov 03 20:13:24.458380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.456683    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7\""
Nov 03 20:13:24.511768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.511710    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"]
Nov 03 20:13:24.521733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.518330    5659 scope.go:122] "RemoveContainer" containerID="dd6324defaae3f0eceddaca8259fba2129d9868bf099d10ee180296fb93b5e1a"
Nov 03 20:13:24.521733 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:24.518739    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6324defaae3f0eceddaca8259fba2129d9868bf099d10ee180296fb93b5e1a\": container with ID starting with dd6324defaae3f0eceddaca8259fba2129d9868bf099d10ee180296fb93b5e1a not found: ID does not exist" containerID="dd6324defaae3f0eceddaca8259fba2129d9868bf099d10ee180296fb93b5e1a"
Nov 03 20:13:24.521733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.518776    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6324defaae3f0eceddaca8259fba2129d9868bf099d10ee180296fb93b5e1a"} err="failed to get container status \"dd6324defaae3f0eceddaca8259fba2129d9868bf099d10ee180296fb93b5e1a\": rpc error: code = NotFound desc = could not find container \"dd6324defaae3f0eceddaca8259fba2129d9868bf099d10ee180296fb93b5e1a\": container with ID starting with dd6324defaae3f0eceddaca8259fba2129d9868bf099d10ee180296fb93b5e1a not found: ID does not exist"
Nov 03 20:13:24.523562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.523533    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-fwmcz"]
Nov 03 20:13:24.527853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.527800    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982" podStartSLOduration=3.52778252 podStartE2EDuration="3.52778252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:24.45957819 +0000 UTC m=+2156.614480610" watchObservedRunningTime="2026-11-03 20:13:24.52778252 +0000 UTC m=+2156.682684941"
Nov 03 20:13:24.615303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.612743    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.615303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.613030    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.615303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.613070    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d62jt\" (UniqueName: \"kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-kube-api-access-d62jt\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.615303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.613176    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.715306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.715266    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.715813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.715791    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.716116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.716091    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.716969 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.716099    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.717079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.717012    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d62jt\" (UniqueName: \"kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-kube-api-access-d62jt\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.717199 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:24.716326    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-98529d/agc-cred-c4c7ed60cafb: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:24.717262 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:24.717207    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:24.717262 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:24.717257    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials podName:600d6fa3-b642-454e-b90a-3f5499770f4b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:26.717241501 +0000 UTC m=+2158.872143901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq" (UID: "600d6fa3-b642-454e-b90a-3f5499770f4b") : secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:24.717866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.717839    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.719937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.719880    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.731588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.730940    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d62jt\" (UniqueName: \"kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-kube-api-access-d62jt\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.822635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.822552    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:24.907121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:24.906165    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"]
Nov 03 20:13:24.909513 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:13:24.909465    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ab3f46_db8e_4d8d_ba17_2841b10cd30b.slice/crio-045c3a25c156705221bb66336262d2a93a16f05c25ed275661decac3154fb740 WatchSource:0}: Error finding container 045c3a25c156705221bb66336262d2a93a16f05c25ed275661decac3154fb740: Status 404 returned error can't find the container with id 045c3a25c156705221bb66336262d2a93a16f05c25ed275661decac3154fb740
Nov 03 20:13:25.050958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.050911    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:25.126900 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.126869    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"]
Nov 03 20:13:25.246678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.246653    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:25.380613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.372711    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" event={"ID":"08ab3f46-db8e-4d8d-ba17-2841b10cd30b","Type":"ContainerStarted","Data":"0f78d1c314a9c5fe90f65d9f568df2980317214c4de4a29c433f17dba46a3839"}
Nov 03 20:13:25.380613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.372809    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" event={"ID":"08ab3f46-db8e-4d8d-ba17-2841b10cd30b","Type":"ContainerStarted","Data":"045c3a25c156705221bb66336262d2a93a16f05c25ed275661decac3154fb740"}
Nov 03 20:13:25.380613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.372882    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:25.380613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.375719    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl" event={"ID":"428e6197-540a-4334-87a8-4a25571a1106","Type":"ContainerStarted","Data":"7f86b20ce3663b29560f4cb0572e632da300ff53617f1111beeb27c6f313d49b"}
Nov 03 20:13:25.380613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.375771    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:25.380613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.375796    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl" event={"ID":"428e6197-540a-4334-87a8-4a25571a1106","Type":"ContainerStarted","Data":"d20534d85afee857044fa453d89392c69e23dd0c0f884178e94b8289f2073917"}
Nov 03 20:13:25.380613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.378239    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv" event={"ID":"229b2907-4221-408f-a677-bfbd3efcae60","Type":"ContainerStarted","Data":"79a0e6e8ce5c389cf692345aee99877d3ba92c42e25b329a075a5c514e563f9d"}
Nov 03 20:13:25.380613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.378312    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv" event={"ID":"229b2907-4221-408f-a677-bfbd3efcae60","Type":"ContainerStarted","Data":"b64cc69364c62a857c77466ab812885f568cba140c368ad6a7adb44af4e4078b"}
Nov 03 20:13:25.380613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.378766    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:25.383862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.383836    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:25.395880 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.395829    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" podStartSLOduration=2.395814698 podStartE2EDuration="2.395814698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:25.393676894 +0000 UTC m=+2157.548579314" watchObservedRunningTime="2026-11-03 20:13:25.395814698 +0000 UTC m=+2157.550717116"
Nov 03 20:13:25.417876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.417809    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl" podStartSLOduration=1.417790066 podStartE2EDuration="1.417790066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:25.415241815 +0000 UTC m=+2157.570144235" watchObservedRunningTime="2026-11-03 20:13:25.417790066 +0000 UTC m=+2157.572692490"
Nov 03 20:13:25.439119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.439061    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv" podStartSLOduration=2.439048568 podStartE2EDuration="2.439048568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:25.434050109 +0000 UTC m=+2157.588952529" watchObservedRunningTime="2026-11-03 20:13:25.439048568 +0000 UTC m=+2157.593950987"
Nov 03 20:13:25.727404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.727362    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"]
Nov 03 20:13:25.835169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:25.835109    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:26.055585 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:26.053680    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:26.081026 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:26.080989    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6671998-ae34-4319-8e68-9785a7d6558d" path="/var/lib/kubelet/pods/a6671998-ae34-4319-8e68-9785a7d6558d/volumes"
Nov 03 20:13:26.425564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:26.421414    5659 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" secret="" err="secret \"payload-processing-external-model-e2e--3d0fafa7-dockercfg-jbrch\" not found"
Nov 03 20:13:26.465996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:26.465950    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"]
Nov 03 20:13:26.511506 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:26.495900    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0e4ce7/agc-cred-1bd75abb0adc: secret "agc-cred-1bd75abb0adc" not found
Nov 03 20:13:26.511506 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:26.495932    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl: secret "agc-cred-1bd75abb0adc" not found
Nov 03 20:13:26.511506 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:26.495990    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials podName:428e6197-540a-4334-87a8-4a25571a1106 nodeName:}" failed. No retries permitted until 2026-11-03 20:13:26.995970836 +0000 UTC m=+2159.150873236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl" (UID: "428e6197-540a-4334-87a8-4a25571a1106") : secret "agc-cred-1bd75abb0adc" not found
Nov 03 20:13:26.511506 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:26.496228    5659 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-plugins-e2e-extmod-a-8b7351: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8b7351" not found
Nov 03 20:13:26.511506 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:26.496278    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume podName:08ab3f46-db8e-4d8d-ba17-2841b10cd30b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:26.99626448 +0000 UTC m=+2159.151166878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" (UID: "08ab3f46-db8e-4d8d-ba17-2841b10cd30b") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8b7351" not found
Nov 03 20:13:26.511506 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:26.505710    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8b7351/agc-cred-5506390946ca: secret "agc-cred-5506390946ca" not found
Nov 03 20:13:26.511506 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:26.505734    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs: secret "agc-cred-5506390946ca" not found
Nov 03 20:13:26.511506 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:26.505793    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials podName:08ab3f46-db8e-4d8d-ba17-2841b10cd30b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:27.005774036 +0000 UTC m=+2159.160676435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" (UID: "08ab3f46-db8e-4d8d-ba17-2841b10cd30b") : secret "agc-cred-5506390946ca" not found
Nov 03 20:13:26.797341 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:26.797308    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:26.802248 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:26.802161    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-98529d/agc-cred-c4c7ed60cafb: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:26.802248 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:26.802197    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:26.802506 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:26.802259    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials podName:600d6fa3-b642-454e-b90a-3f5499770f4b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:30.802238728 +0000 UTC m=+2162.957141128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq" (UID: "600d6fa3-b642-454e-b90a-3f5499770f4b") : secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:27.006199 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:27.006159    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0e4ce7/agc-cred-1bd75abb0adc: secret "agc-cred-1bd75abb0adc" not found
Nov 03 20:13:27.006199 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:27.006204    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl: secret "agc-cred-1bd75abb0adc" not found
Nov 03 20:13:27.006501 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:27.006266    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials podName:428e6197-540a-4334-87a8-4a25571a1106 nodeName:}" failed. No retries permitted until 2026-11-03 20:13:28.006246507 +0000 UTC m=+2160.161148917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl" (UID: "428e6197-540a-4334-87a8-4a25571a1106") : secret "agc-cred-1bd75abb0adc" not found
Nov 03 20:13:27.006501 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:27.006289    5659 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-plugins-e2e-extmod-a-8b7351: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8b7351" not found
Nov 03 20:13:27.006501 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:27.006302    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8b7351/agc-cred-5506390946ca: secret "agc-cred-5506390946ca" not found
Nov 03 20:13:27.006501 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:27.006323    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs: secret "agc-cred-5506390946ca" not found
Nov 03 20:13:27.006501 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:27.006355    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume podName:08ab3f46-db8e-4d8d-ba17-2841b10cd30b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:28.006337009 +0000 UTC m=+2160.161239420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" (UID: "08ab3f46-db8e-4d8d-ba17-2841b10cd30b") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8b7351" not found
Nov 03 20:13:27.006501 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:27.006382    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials podName:08ab3f46-db8e-4d8d-ba17-2841b10cd30b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:28.00637181 +0000 UTC m=+2160.161274213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" (UID: "08ab3f46-db8e-4d8d-ba17-2841b10cd30b") : secret "agc-cred-5506390946ca" not found
Nov 03 20:13:27.554074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:27.554023    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:28.023726 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:28.023694    5659 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-plugins-e2e-extmod-a-8b7351: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8b7351" not found
Nov 03 20:13:28.024156 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:28.023760    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume podName:08ab3f46-db8e-4d8d-ba17-2841b10cd30b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:30.023744335 +0000 UTC m=+2162.178646750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" (UID: "08ab3f46-db8e-4d8d-ba17-2841b10cd30b") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8b7351" not found
Nov 03 20:13:28.024156 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:28.023822    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0e4ce7/agc-cred-1bd75abb0adc: secret "agc-cred-1bd75abb0adc" not found
Nov 03 20:13:28.024156 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:28.023853    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl: secret "agc-cred-1bd75abb0adc" not found
Nov 03 20:13:28.024156 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:28.023909    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials podName:428e6197-540a-4334-87a8-4a25571a1106 nodeName:}" failed. No retries permitted until 2026-11-03 20:13:30.023893315 +0000 UTC m=+2162.178795728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl" (UID: "428e6197-540a-4334-87a8-4a25571a1106") : secret "agc-cred-1bd75abb0adc" not found
Nov 03 20:13:28.024156 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:28.023933    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8b7351/agc-cred-5506390946ca: secret "agc-cred-5506390946ca" not found
Nov 03 20:13:28.024156 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:28.023950    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs: secret "agc-cred-5506390946ca" not found
Nov 03 20:13:28.024156 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:28.023989    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials podName:08ab3f46-db8e-4d8d-ba17-2841b10cd30b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:30.023978929 +0000 UTC m=+2162.178881326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" (UID: "08ab3f46-db8e-4d8d-ba17-2841b10cd30b") : secret "agc-cred-5506390946ca" not found
Nov 03 20:13:28.327534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:28.327434    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:29.149397 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:29.149337    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:29.815964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:29.815927    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:30.119531 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:30.119428    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8b7351/agc-cred-5506390946ca: secret "agc-cred-5506390946ca" not found
Nov 03 20:13:30.119531 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:30.119456    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs: secret "agc-cred-5506390946ca" not found
Nov 03 20:13:30.119531 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:30.119527    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials podName:08ab3f46-db8e-4d8d-ba17-2841b10cd30b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:34.119508136 +0000 UTC m=+2166.274410536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" (UID: "08ab3f46-db8e-4d8d-ba17-2841b10cd30b") : secret "agc-cred-5506390946ca" not found
Nov 03 20:13:30.119878 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:30.119849    5659 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-plugins-e2e-extmod-a-8b7351: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8b7351" not found
Nov 03 20:13:30.120011 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:30.119921    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume podName:08ab3f46-db8e-4d8d-ba17-2841b10cd30b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:34.119907304 +0000 UTC m=+2166.274809709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" (UID: "08ab3f46-db8e-4d8d-ba17-2841b10cd30b") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8b7351" not found
Nov 03 20:13:30.120257 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:30.120231    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0e4ce7/agc-cred-1bd75abb0adc: secret "agc-cred-1bd75abb0adc" not found
Nov 03 20:13:30.120342 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:30.120268    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl: secret "agc-cred-1bd75abb0adc" not found
Nov 03 20:13:30.120342 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:30.120319    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials podName:428e6197-540a-4334-87a8-4a25571a1106 nodeName:}" failed. No retries permitted until 2026-11-03 20:13:34.120302905 +0000 UTC m=+2166.275205306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl" (UID: "428e6197-540a-4334-87a8-4a25571a1106") : secret "agc-cred-1bd75abb0adc" not found
Nov 03 20:13:30.589721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:30.589666    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:30.661504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:30.658413    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:13:30.716768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:30.715593    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"]
Nov 03 20:13:30.833640 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:30.833423    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-98529d/agc-cred-c4c7ed60cafb: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:30.833640 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:30.833452    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:30.833640 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:30.833571    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials podName:600d6fa3-b642-454e-b90a-3f5499770f4b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:38.833551086 +0000 UTC m=+2170.988453499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq" (UID: "600d6fa3-b642-454e-b90a-3f5499770f4b") : secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:31.451190 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:31.451156    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:32.110505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.109806    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:32.251348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.251266    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:32.428507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.428405    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" podUID="08ab3f46-db8e-4d8d-ba17-2841b10cd30b" containerName="payload-processing" containerID="cri-o://0f78d1c314a9c5fe90f65d9f568df2980317214c4de4a29c433f17dba46a3839" gracePeriod=25
Nov 03 20:13:32.428666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.428415    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl" podUID="428e6197-540a-4334-87a8-4a25571a1106" containerName="payload-processing" containerID="cri-o://7f86b20ce3663b29560f4cb0572e632da300ff53617f1111beeb27c6f313d49b" gracePeriod=25
Nov 03 20:13:32.549604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.549345    5659 generic.go:356] "Generic (PLEG): container finished" podID="428e6197-540a-4334-87a8-4a25571a1106" containerID="7f86b20ce3663b29560f4cb0572e632da300ff53617f1111beeb27c6f313d49b" exitCode=0
Nov 03 20:13:32.549604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.549414    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl" event={"ID":"428e6197-540a-4334-87a8-4a25571a1106","Type":"ContainerDied","Data":"7f86b20ce3663b29560f4cb0572e632da300ff53617f1111beeb27c6f313d49b"}
Nov 03 20:13:32.551580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.551467    5659 generic.go:356] "Generic (PLEG): container finished" podID="08ab3f46-db8e-4d8d-ba17-2841b10cd30b" containerID="0f78d1c314a9c5fe90f65d9f568df2980317214c4de4a29c433f17dba46a3839" exitCode=0
Nov 03 20:13:32.551580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.551514    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" event={"ID":"08ab3f46-db8e-4d8d-ba17-2841b10cd30b","Type":"ContainerDied","Data":"0f78d1c314a9c5fe90f65d9f568df2980317214c4de4a29c433f17dba46a3839"}
Nov 03 20:13:32.781174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.781138    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:32.786031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.785943    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:32.901532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.901459    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-routing-overlay-volume\") pod \"428e6197-540a-4334-87a8-4a25571a1106\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") "
Nov 03 20:13:32.901731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.901610    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-kube-api-access-78g98\" (UniqueName: \"kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-kube-api-access-78g98\") pod \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") "
Nov 03 20:13:32.901731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.901640    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-kube-api-access-d62jt\" (UniqueName: \"kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-kube-api-access-d62jt\") pod \"428e6197-540a-4334-87a8-4a25571a1106\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") "
Nov 03 20:13:32.901731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.901665    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-routing-overlay-volume\") pod \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") "
Nov 03 20:13:32.901923 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.901798    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials\") pod \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") "
Nov 03 20:13:32.901923 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.901877    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-routing-overlay-volume" pod "428e6197-540a-4334-87a8-4a25571a1106" (UID: "428e6197-540a-4334-87a8-4a25571a1106"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:32.902036 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.901907    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials\" (UniqueName: \"kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials\") pod \"428e6197-540a-4334-87a8-4a25571a1106\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") "
Nov 03 20:13:32.902036 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.901980    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume\") pod \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\" (UID: \"08ab3f46-db8e-4d8d-ba17-2841b10cd30b\") "
Nov 03 20:13:32.902192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.902173    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-plugins-config-volume\") pod \"428e6197-540a-4334-87a8-4a25571a1106\" (UID: \"428e6197-540a-4334-87a8-4a25571a1106\") "
Nov 03 20:13:32.902312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.902285    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-routing-overlay-volume" pod "08ab3f46-db8e-4d8d-ba17-2841b10cd30b" (UID: "08ab3f46-db8e-4d8d-ba17-2841b10cd30b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:32.902725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.902696    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume" pod "08ab3f46-db8e-4d8d-ba17-2841b10cd30b" (UID: "08ab3f46-db8e-4d8d-ba17-2841b10cd30b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:32.902843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.902729    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-plugins-config-volume" pod "428e6197-540a-4334-87a8-4a25571a1106" (UID: "428e6197-540a-4334-87a8-4a25571a1106"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:32.903188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.903168    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:32.903287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.903192    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:32.903287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.903207    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428e6197-540a-4334-87a8-4a25571a1106-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:32.903287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.903221    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:32.904234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.904212    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-kube-api-access-78g98" pod "08ab3f46-db8e-4d8d-ba17-2841b10cd30b" (UID: "08ab3f46-db8e-4d8d-ba17-2841b10cd30b"). InnerVolumeSpecName "kube-api-access-78g98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:32.904666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.904648    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials" pod "08ab3f46-db8e-4d8d-ba17-2841b10cd30b" (UID: "08ab3f46-db8e-4d8d-ba17-2841b10cd30b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:32.904793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.904778    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials" pod "428e6197-540a-4334-87a8-4a25571a1106" (UID: "428e6197-540a-4334-87a8-4a25571a1106"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:32.904868 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:32.904845    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-kube-api-access-d62jt" pod "428e6197-540a-4334-87a8-4a25571a1106" (UID: "428e6197-540a-4334-87a8-4a25571a1106"). InnerVolumeSpecName "kube-api-access-d62jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:33.004649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.004569    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:33.004649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.004599    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:33.004649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.004608    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78g98\" (UniqueName: \"kubernetes.io/projected/08ab3f46-db8e-4d8d-ba17-2841b10cd30b-kube-api-access-78g98\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:33.004649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.004618    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d62jt\" (UniqueName: \"kubernetes.io/projected/428e6197-540a-4334-87a8-4a25571a1106-kube-api-access-d62jt\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:33.154113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.154075    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:33.563846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.563805    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"
Nov 03 20:13:33.563846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.563832    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl" event={"ID":"428e6197-540a-4334-87a8-4a25571a1106","Type":"ContainerDied","Data":"d20534d85afee857044fa453d89392c69e23dd0c0f884178e94b8289f2073917"}
Nov 03 20:13:33.564116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.563898    5659 scope.go:122] "RemoveContainer" containerID="7f86b20ce3663b29560f4cb0572e632da300ff53617f1111beeb27c6f313d49b"
Nov 03 20:13:33.565783 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.565757    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"
Nov 03 20:13:33.565783 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.565765    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs" event={"ID":"08ab3f46-db8e-4d8d-ba17-2841b10cd30b","Type":"ContainerDied","Data":"045c3a25c156705221bb66336262d2a93a16f05c25ed275661decac3154fb740"}
Nov 03 20:13:33.577981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.577957    5659 scope.go:122] "RemoveContainer" containerID="0f78d1c314a9c5fe90f65d9f568df2980317214c4de4a29c433f17dba46a3839"
Nov 03 20:13:33.596221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.596177    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"]
Nov 03 20:13:33.601076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.601050    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-77969gjjs"]
Nov 03 20:13:33.613094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.613069    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"]
Nov 03 20:13:33.620704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.620676    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-69c52hxtl"]
Nov 03 20:13:33.829914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.829831    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:33.971810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.971778    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ab3f46-db8e-4d8d-ba17-2841b10cd30b" path="/var/lib/kubelet/pods/08ab3f46-db8e-4d8d-ba17-2841b10cd30b/volumes"
Nov 03 20:13:33.972302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:33.972287    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428e6197-540a-4334-87a8-4a25571a1106" path="/var/lib/kubelet/pods/428e6197-540a-4334-87a8-4a25571a1106/volumes"
Nov 03 20:13:34.568110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:34.568069    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:35.362503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:35.361812    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:35.396226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:35.396183    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:13:35.403108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:35.403081    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:35.716916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:35.716881    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2" podUID="9f0a30ee-0be2-4abf-8624-d76f418d104f" containerName="payload-processing" containerID="cri-o://2e2d472a12057d9aa721fa893abf8db175995e2d4529ba61107c898d55e9e90d" gracePeriod=25
Nov 03 20:13:36.030149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.030124    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:13:36.035166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.035134    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:36.173887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.173749    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-routing-overlay-volume\") pod \"9f0a30ee-0be2-4abf-8624-d76f418d104f\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") "
Nov 03 20:13:36.173887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.173870    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-kube-api-access-dld94\" (UniqueName: \"kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-kube-api-access-dld94\") pod \"9f0a30ee-0be2-4abf-8624-d76f418d104f\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") "
Nov 03 20:13:36.174354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.174099    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-provider-credentials\") pod \"9f0a30ee-0be2-4abf-8624-d76f418d104f\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") "
Nov 03 20:13:36.174354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.174181    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-routing-overlay-volume" pod "9f0a30ee-0be2-4abf-8624-d76f418d104f" (UID: "9f0a30ee-0be2-4abf-8624-d76f418d104f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:36.174354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.174233    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-plugins-config-volume\") pod \"9f0a30ee-0be2-4abf-8624-d76f418d104f\" (UID: \"9f0a30ee-0be2-4abf-8624-d76f418d104f\") "
Nov 03 20:13:36.175209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.175175    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:36.175428 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.175400    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-plugins-config-volume" pod "9f0a30ee-0be2-4abf-8624-d76f418d104f" (UID: "9f0a30ee-0be2-4abf-8624-d76f418d104f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:36.177240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.177212    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-provider-credentials" pod "9f0a30ee-0be2-4abf-8624-d76f418d104f" (UID: "9f0a30ee-0be2-4abf-8624-d76f418d104f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:36.177511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.177466    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-kube-api-access-dld94" pod "9f0a30ee-0be2-4abf-8624-d76f418d104f" (UID: "9f0a30ee-0be2-4abf-8624-d76f418d104f"). InnerVolumeSpecName "kube-api-access-dld94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:36.277584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.277475    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0a30ee-0be2-4abf-8624-d76f418d104f-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:36.277584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.277523    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dld94\" (UniqueName: \"kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-kube-api-access-dld94\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:36.277584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.277533    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f0a30ee-0be2-4abf-8624-d76f418d104f-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:36.424869 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.424834    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:13:36.496575 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.496540    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"]
Nov 03 20:13:36.639713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.639624    5659 generic.go:356] "Generic (PLEG): container finished" podID="9f0a30ee-0be2-4abf-8624-d76f418d104f" containerID="2e2d472a12057d9aa721fa893abf8db175995e2d4529ba61107c898d55e9e90d" exitCode=0
Nov 03 20:13:36.639860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.639708    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"
Nov 03 20:13:36.639860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.639707    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2" event={"ID":"9f0a30ee-0be2-4abf-8624-d76f418d104f","Type":"ContainerDied","Data":"2e2d472a12057d9aa721fa893abf8db175995e2d4529ba61107c898d55e9e90d"}
Nov 03 20:13:36.639860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.639829    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2" event={"ID":"9f0a30ee-0be2-4abf-8624-d76f418d104f","Type":"ContainerDied","Data":"652608ae475a22b22223c3d5596d0c0cc14c0ebddf98d3bf648e221334d35a26"}
Nov 03 20:13:36.639860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.639854    5659 scope.go:122] "RemoveContainer" containerID="2e2d472a12057d9aa721fa893abf8db175995e2d4529ba61107c898d55e9e90d"
Nov 03 20:13:36.651650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.651628    5659 scope.go:122] "RemoveContainer" containerID="2e2d472a12057d9aa721fa893abf8db175995e2d4529ba61107c898d55e9e90d"
Nov 03 20:13:36.651960 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:36.651930    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2d472a12057d9aa721fa893abf8db175995e2d4529ba61107c898d55e9e90d\": container with ID starting with 2e2d472a12057d9aa721fa893abf8db175995e2d4529ba61107c898d55e9e90d not found: ID does not exist" containerID="2e2d472a12057d9aa721fa893abf8db175995e2d4529ba61107c898d55e9e90d"
Nov 03 20:13:36.652034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.651973    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2d472a12057d9aa721fa893abf8db175995e2d4529ba61107c898d55e9e90d"} err="failed to get container status \"2e2d472a12057d9aa721fa893abf8db175995e2d4529ba61107c898d55e9e90d\": rpc error: code = NotFound desc = could not find container \"2e2d472a12057d9aa721fa893abf8db175995e2d4529ba61107c898d55e9e90d\": container with ID starting with 2e2d472a12057d9aa721fa893abf8db175995e2d4529ba61107c898d55e9e90d not found: ID does not exist"
Nov 03 20:13:36.671746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.671712    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"]
Nov 03 20:13:36.678372 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.678346    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-54fcz27r2"]
Nov 03 20:13:36.841849 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:36.841805    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:37.635854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:37.635823    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:37.820071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:37.820040    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:37.972110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:37.972071    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f0a30ee-0be2-4abf-8624-d76f418d104f" path="/var/lib/kubelet/pods/9f0a30ee-0be2-4abf-8624-d76f418d104f/volumes"
Nov 03 20:13:38.317757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.317677    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:38.561051 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.561019    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:38.704260 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.704228    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"]
Nov 03 20:13:38.705600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.705568    5659 cpu_manager.go:414] "containerMap: removing container" podUID="9f0a30ee-0be2-4abf-8624-d76f418d104f" containerName="payload-processing"
Nov 03 20:13:38.705600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.705597    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0a30ee-0be2-4abf-8624-d76f418d104f" containerName="payload-processing"
Nov 03 20:13:38.705787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.705643    5659 cpu_manager.go:414] "containerMap: removing container" podUID="08ab3f46-db8e-4d8d-ba17-2841b10cd30b" containerName="payload-processing"
Nov 03 20:13:38.705787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.705652    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ab3f46-db8e-4d8d-ba17-2841b10cd30b" containerName="payload-processing"
Nov 03 20:13:38.705787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.705698    5659 cpu_manager.go:414] "containerMap: removing container" podUID="428e6197-540a-4334-87a8-4a25571a1106" containerName="payload-processing"
Nov 03 20:13:38.705787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.705706    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428e6197-540a-4334-87a8-4a25571a1106" containerName="payload-processing"
Nov 03 20:13:38.706035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.705866    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="428e6197-540a-4334-87a8-4a25571a1106" containerName="payload-processing"
Nov 03 20:13:38.706035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.705884    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0a30ee-0be2-4abf-8624-d76f418d104f" containerName="payload-processing"
Nov 03 20:13:38.706035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.705902    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ab3f46-db8e-4d8d-ba17-2841b10cd30b" containerName="payload-processing"
Nov 03 20:13:38.711053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.711032    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.718049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.718026    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b503a8-openshift-default-dockercfg-tchtj\""
Nov 03 20:13:38.720282 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.720257    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"]
Nov 03 20:13:38.820925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.820844    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-envoy\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.820925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.820895    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.821135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.821023    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-credential-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.821135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.821055    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-token\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.821135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.821096    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxf62\" (UniqueName: \"kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-kube-api-access-nxf62\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.821234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.821184    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-data\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.821234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.821217    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-podinfo\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.821471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.821451    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1181f6f2-8a83-4b5f-bf64-77a556de321d-istiod-ca-cert\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.821561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.821547    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-certs\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.922708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.922655    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.922862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.922757    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-credential-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.922862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.922786    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-token\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.922862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.922842    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxf62\" (UniqueName: \"kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-kube-api-access-nxf62\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.923061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.922887    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-data\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.923061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.922920    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-podinfo\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.923176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.923121    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.923237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.923140    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-credential-socket\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.923237 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:38.923202    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-98529d/agc-cred-c4c7ed60cafb: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:38.923237 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:38.923224    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:38.923407 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:38.923315    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials podName:600d6fa3-b642-454e-b90a-3f5499770f4b nodeName:}" failed. No retries permitted until 2026-11-03 20:13:54.923261697 +0000 UTC m=+2187.078164094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq" (UID: "600d6fa3-b642-454e-b90a-3f5499770f4b") : secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:13:38.923407 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.923348    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1181f6f2-8a83-4b5f-bf64-77a556de321d-istiod-ca-cert\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.923407 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.923369    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-data\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.923407 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.923403    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-certs\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.923669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.923505    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-envoy\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.923771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.923744    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-certs\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.924160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.924139    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1181f6f2-8a83-4b5f-bf64-77a556de321d-istiod-ca-cert\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.925862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.925839    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-envoy\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.925948 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.925840    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-podinfo\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.934826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.934798    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-token\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:38.935278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:38.935254    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxf62\" (UniqueName: \"kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-kube-api-access-nxf62\") pod \"e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") " pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:39.030571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.030442    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:39.031399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.031377    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:39.397031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.396998    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"]
Nov 03 20:13:39.399705 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:13:39.399670    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1181f6f2_8a83_4b5f_bf64_77a556de321d.slice/crio-95144b6cbb2ab16e63d8ea80d702e5a1ee260d48629451963bb392c204acddc2 WatchSource:0}: Error finding container 95144b6cbb2ab16e63d8ea80d702e5a1ee260d48629451963bb392c204acddc2: Status 404 returned error can't find the container with id 95144b6cbb2ab16e63d8ea80d702e5a1ee260d48629451963bb392c204acddc2
Nov 03 20:13:39.402609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.402574    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:13:39.402729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.402641    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:13:39.402729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.402681    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:13:39.519425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.519393    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:39.681559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.674082    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:39.744597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.739856    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"]
Nov 03 20:13:39.754572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.746517    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:39.773505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.772952    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"kube-root-ca.crt\""
Nov 03 20:13:39.773505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.773205    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"agc-cred-6e3d6ce11b17\""
Nov 03 20:13:39.777511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.773597    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-e2e--b1878920-dockercfg-6sqzm\""
Nov 03 20:13:39.790561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.789816    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"routing-overlay\""
Nov 03 20:13:39.795846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.795801    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7\""
Nov 03 20:13:39.811886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.811853    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"]
Nov 03 20:13:39.821199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.821166    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"openshift-service-ca.crt\""
Nov 03 20:13:39.834883 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.834844    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt" event={"ID":"1181f6f2-8a83-4b5f-bf64-77a556de321d","Type":"ContainerStarted","Data":"d96f5ec7128d6df2a716e894e3a4facc7b2e0200e28ef9e52099f97d237a6bdc"}
Nov 03 20:13:39.835031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.834917    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt" event={"ID":"1181f6f2-8a83-4b5f-bf64-77a556de321d","Type":"ContainerStarted","Data":"95144b6cbb2ab16e63d8ea80d702e5a1ee260d48629451963bb392c204acddc2"}
Nov 03 20:13:39.840944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.840910    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:39.841110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.840975    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:39.841247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.841218    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvbm4\" (UniqueName: \"kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-kube-api-access-cvbm4\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:39.841420 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.841396    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:39.861928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.861863    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt" podStartSLOduration=1.8618430350000001 podStartE2EDuration="1.861843035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:39.858446404 +0000 UTC m=+2172.013348880" watchObservedRunningTime="2026-11-03 20:13:39.861843035 +0000 UTC m=+2172.016745456"
Nov 03 20:13:39.943156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.943073    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:39.943367 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.943344    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:39.943446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.943406    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:39.943587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.943566    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvbm4\" (UniqueName: \"kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-kube-api-access-cvbm4\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:39.944046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.944015    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:39.944297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.944270    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:39.945879 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.945859    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:39.956355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:39.956332    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvbm4\" (UniqueName: \"kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-kube-api-access-cvbm4\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:40.049505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:40.049074    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:40.060504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:40.054592    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:13:40.060504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:40.054650    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt" podUID="1181f6f2-8a83-4b5f-bf64-77a556de321d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 20:13:40.134089 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:40.133797    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:40.498126 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:40.498033    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:40.683755 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:13:40.682964    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1abda46_8fdd_43fe_9a79_889d9bc2d38b.slice/crio-4cfd14b112fe55c29f64ee69eec6c76359525b8a1050b837a1cdc8e468d60212 WatchSource:0}: Error finding container 4cfd14b112fe55c29f64ee69eec6c76359525b8a1050b837a1cdc8e468d60212: Status 404 returned error can't find the container with id 4cfd14b112fe55c29f64ee69eec6c76359525b8a1050b837a1cdc8e468d60212
Nov 03 20:13:40.693671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:40.692965    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"]
Nov 03 20:13:40.869254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:40.869217    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt" event={"ID":"d1abda46-8fdd-43fe-9a79-889d9bc2d38b","Type":"ContainerStarted","Data":"b6176abc3719f51ebfac6074414ba3de07afaa7c383c4b6a0fba207aa1714d1c"}
Nov 03 20:13:40.869424 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:40.869266    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt" event={"ID":"d1abda46-8fdd-43fe-9a79-889d9bc2d38b","Type":"ContainerStarted","Data":"4cfd14b112fe55c29f64ee69eec6c76359525b8a1050b837a1cdc8e468d60212"}
Nov 03 20:13:40.869424 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:40.869411    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:40.893692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:40.893646    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt" podStartSLOduration=1.8936331119999998 podStartE2EDuration="1.893633112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:40.889587177 +0000 UTC m=+2173.044489599" watchObservedRunningTime="2026-11-03 20:13:40.893633112 +0000 UTC m=+2173.048535527"
Nov 03 20:13:41.042565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:41.035948    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:13:41.042565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:41.036013    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt" podUID="1181f6f2-8a83-4b5f-bf64-77a556de321d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 20:13:41.365818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:41.365017    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:41.497176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:41.497136    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq" podUID="600d6fa3-b642-454e-b90a-3f5499770f4b" containerName="payload-processing" containerID="cri-o://942fcc710d5a2905d4afb412ddff6accaabb6b6e9f714d7eafb6d91e72aaeaf5" gracePeriod=25
Nov 03 20:13:41.865760 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:41.865731    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:41.878190 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:41.878160    5659 generic.go:356] "Generic (PLEG): container finished" podID="600d6fa3-b642-454e-b90a-3f5499770f4b" containerID="942fcc710d5a2905d4afb412ddff6accaabb6b6e9f714d7eafb6d91e72aaeaf5" exitCode=0
Nov 03 20:13:41.878334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:41.878196    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"
Nov 03 20:13:41.878334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:41.878233    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq" event={"ID":"600d6fa3-b642-454e-b90a-3f5499770f4b","Type":"ContainerDied","Data":"942fcc710d5a2905d4afb412ddff6accaabb6b6e9f714d7eafb6d91e72aaeaf5"}
Nov 03 20:13:41.878334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:41.878281    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq" event={"ID":"600d6fa3-b642-454e-b90a-3f5499770f4b","Type":"ContainerDied","Data":"1f7062bb34a8f777b5fe896d0cd83273765d0f198606e0264409946493cd48be"}
Nov 03 20:13:41.878334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:41.878307    5659 scope.go:122] "RemoveContainer" containerID="942fcc710d5a2905d4afb412ddff6accaabb6b6e9f714d7eafb6d91e72aaeaf5"
Nov 03 20:13:41.891331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:41.891278    5659 scope.go:122] "RemoveContainer" containerID="942fcc710d5a2905d4afb412ddff6accaabb6b6e9f714d7eafb6d91e72aaeaf5"
Nov 03 20:13:41.892012 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:41.891982    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"942fcc710d5a2905d4afb412ddff6accaabb6b6e9f714d7eafb6d91e72aaeaf5\": container with ID starting with 942fcc710d5a2905d4afb412ddff6accaabb6b6e9f714d7eafb6d91e72aaeaf5 not found: ID does not exist" containerID="942fcc710d5a2905d4afb412ddff6accaabb6b6e9f714d7eafb6d91e72aaeaf5"
Nov 03 20:13:41.892133 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:41.892024    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"942fcc710d5a2905d4afb412ddff6accaabb6b6e9f714d7eafb6d91e72aaeaf5"} err="failed to get container status \"942fcc710d5a2905d4afb412ddff6accaabb6b6e9f714d7eafb6d91e72aaeaf5\": rpc error: code = NotFound desc = could not find container \"942fcc710d5a2905d4afb412ddff6accaabb6b6e9f714d7eafb6d91e72aaeaf5\": container with ID starting with 942fcc710d5a2905d4afb412ddff6accaabb6b6e9f714d7eafb6d91e72aaeaf5 not found: ID does not exist"
Nov 03 20:13:42.022619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.021113    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials\") pod \"600d6fa3-b642-454e-b90a-3f5499770f4b\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") "
Nov 03 20:13:42.022619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.021187    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-kube-api-access-2qphj\" (UniqueName: \"kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-kube-api-access-2qphj\") pod \"600d6fa3-b642-454e-b90a-3f5499770f4b\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") "
Nov 03 20:13:42.022619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.021228    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-plugins-config-volume\") pod \"600d6fa3-b642-454e-b90a-3f5499770f4b\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") "
Nov 03 20:13:42.022619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.021259    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-routing-overlay-volume\") pod \"600d6fa3-b642-454e-b90a-3f5499770f4b\" (UID: \"600d6fa3-b642-454e-b90a-3f5499770f4b\") "
Nov 03 20:13:42.023191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.023161    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-routing-overlay-volume" pod "600d6fa3-b642-454e-b90a-3f5499770f4b" (UID: "600d6fa3-b642-454e-b90a-3f5499770f4b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:42.023661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.023638    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-plugins-config-volume" pod "600d6fa3-b642-454e-b90a-3f5499770f4b" (UID: "600d6fa3-b642-454e-b90a-3f5499770f4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:42.027347 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.027316    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-kube-api-access-2qphj" pod "600d6fa3-b642-454e-b90a-3f5499770f4b" (UID: "600d6fa3-b642-454e-b90a-3f5499770f4b"). InnerVolumeSpecName "kube-api-access-2qphj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:42.027526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.027506    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials" pod "600d6fa3-b642-454e-b90a-3f5499770f4b" (UID: "600d6fa3-b642-454e-b90a-3f5499770f4b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:42.055728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.054900    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:13:42.055728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.054957    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt" podUID="1181f6f2-8a83-4b5f-bf64-77a556de321d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 20:13:42.123551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.123507    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qphj\" (UniqueName: \"kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-kube-api-access-2qphj\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:42.123551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.123541    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:42.123775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.123556    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/600d6fa3-b642-454e-b90a-3f5499770f4b-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:42.123775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.123572    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/600d6fa3-b642-454e-b90a-3f5499770f4b-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:42.185701 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.185667    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:42.256620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.256539    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"]
Nov 03 20:13:42.301946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:42.301858    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hcfqq"]
Nov 03 20:13:43.036108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:43.036063    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:13:43.036729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:43.036695    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt" podUID="1181f6f2-8a83-4b5f-bf64-77a556de321d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 20:13:43.182806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:43.182762    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:43.719511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:43.719000    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:43.869522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:43.862891    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:44.045564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:44.031879    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:13:44.045564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:44.031938    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt" podUID="1181f6f2-8a83-4b5f-bf64-77a556de321d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 20:13:44.053555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:44.051908    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600d6fa3-b642-454e-b90a-3f5499770f4b" path="/var/lib/kubelet/pods/600d6fa3-b642-454e-b90a-3f5499770f4b/volumes"
Nov 03 20:13:44.356605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:44.356466    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:44.866254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:44.866178    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:45.050516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:45.049937    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:13:45.050516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:45.050003    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt" podUID="1181f6f2-8a83-4b5f-bf64-77a556de321d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 20:13:45.218776 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:45.218735    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:13:45.565623 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:45.564001    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:46.040629 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:46.040590    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:13:46.040920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:46.040889    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt" podUID="1181f6f2-8a83-4b5f-bf64-77a556de321d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 20:13:46.367556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:46.347680    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:47.038113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:47.038081    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:47.038534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:47.038508    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:47.039242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:47.039224    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:13:47.289559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:47.278629    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:48.031502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:48.031460    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:48.839678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:48.839644    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:49.644519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:49.638399    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:50.360665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:50.360620    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:50.771220 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:50.771182    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:50.902558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:50.902525    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:51.190439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:51.190365    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:51.281075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:51.281032    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"]
Nov 03 20:13:51.284870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:51.284845    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:51.328330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:51.328295    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:51.499018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:51.498978    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:51.923540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:51.923444    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:52.342967 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:52.342932    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:53.053849 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:53.053817    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:53.736785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:53.736731    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:54.445464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:54.445431    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:55.293375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:55.293341    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:56.026013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:56.025967    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:56.282568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:56.282445    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt" podUID="d1abda46-8fdd-43fe-9a79-889d9bc2d38b" containerName="payload-processing" containerID="cri-o://b6176abc3719f51ebfac6074414ba3de07afaa7c383c4b6a0fba207aa1714d1c" gracePeriod=25
Nov 03 20:13:56.874284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:56.874250    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:57.036822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.036785    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-provider-credentials\") pod \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") "
Nov 03 20:13:57.048454 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.048417    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-routing-overlay-volume\") pod \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") "
Nov 03 20:13:57.048671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.048506    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-plugins-config-volume\") pod \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") "
Nov 03 20:13:57.048915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.048886    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-kube-api-access-cvbm4\" (UniqueName: \"kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-kube-api-access-cvbm4\") pod \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\" (UID: \"d1abda46-8fdd-43fe-9a79-889d9bc2d38b\") "
Nov 03 20:13:57.051821 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.051108    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-plugins-config-volume" pod "d1abda46-8fdd-43fe-9a79-889d9bc2d38b" (UID: "d1abda46-8fdd-43fe-9a79-889d9bc2d38b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:57.064792 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.064754    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:57.065135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.065113    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-routing-overlay-volume" pod "d1abda46-8fdd-43fe-9a79-889d9bc2d38b" (UID: "d1abda46-8fdd-43fe-9a79-889d9bc2d38b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:57.084707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.083901    5659 generic.go:356] "Generic (PLEG): container finished" podID="d1abda46-8fdd-43fe-9a79-889d9bc2d38b" containerID="b6176abc3719f51ebfac6074414ba3de07afaa7c383c4b6a0fba207aa1714d1c" exitCode=0
Nov 03 20:13:57.084707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.083999    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt" event={"ID":"d1abda46-8fdd-43fe-9a79-889d9bc2d38b","Type":"ContainerDied","Data":"b6176abc3719f51ebfac6074414ba3de07afaa7c383c4b6a0fba207aa1714d1c"}
Nov 03 20:13:57.084707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.084052    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt" event={"ID":"d1abda46-8fdd-43fe-9a79-889d9bc2d38b","Type":"ContainerDied","Data":"4cfd14b112fe55c29f64ee69eec6c76359525b8a1050b837a1cdc8e468d60212"}
Nov 03 20:13:57.084707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.084082    5659 scope.go:122] "RemoveContainer" containerID="b6176abc3719f51ebfac6074414ba3de07afaa7c383c4b6a0fba207aa1714d1c"
Nov 03 20:13:57.084707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.084243    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"
Nov 03 20:13:57.183514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.168038    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-provider-credentials" pod "d1abda46-8fdd-43fe-9a79-889d9bc2d38b" (UID: "d1abda46-8fdd-43fe-9a79-889d9bc2d38b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:57.183514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.168976    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-kube-api-access-cvbm4" pod "d1abda46-8fdd-43fe-9a79-889d9bc2d38b" (UID: "d1abda46-8fdd-43fe-9a79-889d9bc2d38b"). InnerVolumeSpecName "kube-api-access-cvbm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:57.207978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.184227    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:57.207978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.184266    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:57.207978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.184281    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvbm4\" (UniqueName: \"kubernetes.io/projected/d1abda46-8fdd-43fe-9a79-889d9bc2d38b-kube-api-access-cvbm4\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:57.333380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.332642    5659 scope.go:122] "RemoveContainer" containerID="b6176abc3719f51ebfac6074414ba3de07afaa7c383c4b6a0fba207aa1714d1c"
Nov 03 20:13:57.333916 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:13:57.333882    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6176abc3719f51ebfac6074414ba3de07afaa7c383c4b6a0fba207aa1714d1c\": container with ID starting with b6176abc3719f51ebfac6074414ba3de07afaa7c383c4b6a0fba207aa1714d1c not found: ID does not exist" containerID="b6176abc3719f51ebfac6074414ba3de07afaa7c383c4b6a0fba207aa1714d1c"
Nov 03 20:13:57.334109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.334077    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6176abc3719f51ebfac6074414ba3de07afaa7c383c4b6a0fba207aa1714d1c"} err="failed to get container status \"b6176abc3719f51ebfac6074414ba3de07afaa7c383c4b6a0fba207aa1714d1c\": rpc error: code = NotFound desc = could not find container \"b6176abc3719f51ebfac6074414ba3de07afaa7c383c4b6a0fba207aa1714d1c\": container with ID starting with b6176abc3719f51ebfac6074414ba3de07afaa7c383c4b6a0fba207aa1714d1c not found: ID does not exist"
Nov 03 20:13:57.385186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.385149    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:57.497371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.493382    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"]
Nov 03 20:13:57.503426 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.503392    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbbdgzgt"]
Nov 03 20:13:57.804861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.804827    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:57.874190 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.874149    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"]
Nov 03 20:13:57.933202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.933152    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:57.974055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:57.974013    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1abda46-8fdd-43fe-9a79-889d9bc2d38b" path="/var/lib/kubelet/pods/d1abda46-8fdd-43fe-9a79-889d9bc2d38b/volumes"
Nov 03 20:13:58.459443 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:58.459409    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:58.614079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:58.614042    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:58.764353 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:58.764268    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:58.919478 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:58.919443    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:59.055446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:59.055365    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:59.854785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:13:59.854749    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:00.697508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:00.696192    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:00.932228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:00.932199    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:01.458205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:01.458170    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:02.155521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:02.155467    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:02.824049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:02.824017    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:02.885522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:02.876806    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg" podUID="bc9b00f9-5c52-4a44-8f8e-96a5f8408118" containerName="payload-processing" containerID="cri-o://9ee2d13554e2b96203d31137ff42dfce6ff2ab9a324e854c7608846ae9c3d1b4" gracePeriod=25
Nov 03 20:14:03.214688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.214657    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:14:03.217075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.217049    5659 generic.go:356] "Generic (PLEG): container finished" podID="bc9b00f9-5c52-4a44-8f8e-96a5f8408118" containerID="9ee2d13554e2b96203d31137ff42dfce6ff2ab9a324e854c7608846ae9c3d1b4" exitCode=0
Nov 03 20:14:03.217221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.217096    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"
Nov 03 20:14:03.217221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.217127    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg" event={"ID":"bc9b00f9-5c52-4a44-8f8e-96a5f8408118","Type":"ContainerDied","Data":"9ee2d13554e2b96203d31137ff42dfce6ff2ab9a324e854c7608846ae9c3d1b4"}
Nov 03 20:14:03.217221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.217202    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg" event={"ID":"bc9b00f9-5c52-4a44-8f8e-96a5f8408118","Type":"ContainerDied","Data":"30d647c139e1599885857549963ae2fec620cd83aba1d96accbf6b477e3ac733"}
Nov 03 20:14:03.217221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.217212    5659 scope.go:122] "RemoveContainer" containerID="9ee2d13554e2b96203d31137ff42dfce6ff2ab9a324e854c7608846ae9c3d1b4"
Nov 03 20:14:03.228005 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.227988    5659 scope.go:122] "RemoveContainer" containerID="9ee2d13554e2b96203d31137ff42dfce6ff2ab9a324e854c7608846ae9c3d1b4"
Nov 03 20:14:03.228307 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:14:03.228286    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee2d13554e2b96203d31137ff42dfce6ff2ab9a324e854c7608846ae9c3d1b4\": container with ID starting with 9ee2d13554e2b96203d31137ff42dfce6ff2ab9a324e854c7608846ae9c3d1b4 not found: ID does not exist" containerID="9ee2d13554e2b96203d31137ff42dfce6ff2ab9a324e854c7608846ae9c3d1b4"
Nov 03 20:14:03.228365 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.228316    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee2d13554e2b96203d31137ff42dfce6ff2ab9a324e854c7608846ae9c3d1b4"} err="failed to get container status \"9ee2d13554e2b96203d31137ff42dfce6ff2ab9a324e854c7608846ae9c3d1b4\": rpc error: code = NotFound desc = could not find container \"9ee2d13554e2b96203d31137ff42dfce6ff2ab9a324e854c7608846ae9c3d1b4\": container with ID starting with 9ee2d13554e2b96203d31137ff42dfce6ff2ab9a324e854c7608846ae9c3d1b4 not found: ID does not exist"
Nov 03 20:14:03.343032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.342929    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-provider-credentials\") pod \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") "
Nov 03 20:14:03.343032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.343031    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-kube-api-access-gfkfg\" (UniqueName: \"kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-kube-api-access-gfkfg\") pod \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") "
Nov 03 20:14:03.343292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.343119    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-routing-overlay-volume\") pod \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") "
Nov 03 20:14:03.343747 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.343402    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-plugins-config-volume\") pod \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\" (UID: \"bc9b00f9-5c52-4a44-8f8e-96a5f8408118\") "
Nov 03 20:14:03.343747 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.343644    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-routing-overlay-volume" pod "bc9b00f9-5c52-4a44-8f8e-96a5f8408118" (UID: "bc9b00f9-5c52-4a44-8f8e-96a5f8408118"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:14:03.344184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.344153    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-plugins-config-volume" pod "bc9b00f9-5c52-4a44-8f8e-96a5f8408118" (UID: "bc9b00f9-5c52-4a44-8f8e-96a5f8408118"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:14:03.344270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.344157    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:03.345496 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.345461    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-provider-credentials" pod "bc9b00f9-5c52-4a44-8f8e-96a5f8408118" (UID: "bc9b00f9-5c52-4a44-8f8e-96a5f8408118"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:03.345725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.345700    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-kube-api-access-gfkfg" pod "bc9b00f9-5c52-4a44-8f8e-96a5f8408118" (UID: "bc9b00f9-5c52-4a44-8f8e-96a5f8408118"). InnerVolumeSpecName "kube-api-access-gfkfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:03.446272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.446228    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:03.446272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.446268    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:03.446272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.446277    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfkfg\" (UniqueName: \"kubernetes.io/projected/bc9b00f9-5c52-4a44-8f8e-96a5f8408118-kube-api-access-gfkfg\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:03.544702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.544653    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"]
Nov 03 20:14:03.549113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.549079    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-b7fc4f9fg"]
Nov 03 20:14:03.565745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.565712    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:03.970439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:03.970404    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9b00f9-5c52-4a44-8f8e-96a5f8408118" path="/var/lib/kubelet/pods/bc9b00f9-5c52-4a44-8f8e-96a5f8408118/volumes"
Nov 03 20:14:04.378797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:04.378716    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:05.139158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:05.139128    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:05.304189 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:05.304160    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:05.573165 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:05.573127    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:05.739964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:05.739931    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:05.880122 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:05.880019    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:06.023028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:06.022994    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:06.300782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:06.300750    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:06.442558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:06.442525    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:06.902809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:06.902776    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:07.047000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.046964    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:07.497611 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.497580    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:07.603738 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.603703    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"]
Nov 03 20:14:07.604467 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.604453    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bc9b00f9-5c52-4a44-8f8e-96a5f8408118" containerName="payload-processing"
Nov 03 20:14:07.604550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.604469    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9b00f9-5c52-4a44-8f8e-96a5f8408118" containerName="payload-processing"
Nov 03 20:14:07.604550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.604502    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d1abda46-8fdd-43fe-9a79-889d9bc2d38b" containerName="payload-processing"
Nov 03 20:14:07.604550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.604508    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1abda46-8fdd-43fe-9a79-889d9bc2d38b" containerName="payload-processing"
Nov 03 20:14:07.604550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.604550    5659 cpu_manager.go:414] "containerMap: removing container" podUID="600d6fa3-b642-454e-b90a-3f5499770f4b" containerName="payload-processing"
Nov 03 20:14:07.604757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.604555    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600d6fa3-b642-454e-b90a-3f5499770f4b" containerName="payload-processing"
Nov 03 20:14:07.604757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.604667    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1abda46-8fdd-43fe-9a79-889d9bc2d38b" containerName="payload-processing"
Nov 03 20:14:07.604757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.604683    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="600d6fa3-b642-454e-b90a-3f5499770f4b" containerName="payload-processing"
Nov 03 20:14:07.604757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.604693    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9b00f9-5c52-4a44-8f8e-96a5f8408118" containerName="payload-processing"
Nov 03 20:14:07.608654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.608628    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:07.613850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.613828    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"agc-cred-a4e8a8a359a4\""
Nov 03 20:14:07.614441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.614421    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"payload-processing-external-model-e2e--3d0fafa7-dockercfg-mk2qh\""
Nov 03 20:14:07.629221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.629191    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"]
Nov 03 20:14:07.651642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.651610    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:07.675204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.675047    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"routing-overlay\""
Nov 03 20:14:07.675204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.675058    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"openshift-service-ca.crt\""
Nov 03 20:14:07.675204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.675170    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8b7351\""
Nov 03 20:14:07.675518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.675230    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"kube-root-ca.crt\""
Nov 03 20:14:07.679146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.678633    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ttvk\" (UniqueName: \"kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-kube-api-access-8ttvk\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:07.679146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.678682    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:07.679146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.678712    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:07.679146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.678761    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:07.830553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.830261    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:07.854509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.849267    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ttvk\" (UniqueName: \"kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-kube-api-access-8ttvk\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:07.988524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.876059    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:08.020154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.898504    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:08.042262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.958583    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:08.042262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:07.979227    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ttvk\" (UniqueName: \"kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-kube-api-access-8ttvk\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:08.065545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:08.063571    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:08.065545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:08.063648    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:08.065545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:08.064674    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:08.213372 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:08.207384    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:08.222284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:08.222253    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:08.694163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:08.692861    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"]
Nov 03 20:14:08.718942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:08.697998    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:08.930415 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:14:08.924616    5659 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b57093_ed30_4994_bfc5_2f6901e467b9.slice/crio-045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:14:09.019219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.019166    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:09.379511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.377426    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895" event={"ID":"61b57093-ed30-4994-bfc5-2f6901e467b9","Type":"ContainerStarted","Data":"045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513"}
Nov 03 20:14:09.379511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.377514    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895" event={"ID":"61b57093-ed30-4994-bfc5-2f6901e467b9","Type":"ContainerStarted","Data":"51a1dfce39cdfd3ea10cb839955e1c4d02164a3580f3e2326df7672e8aa0f402"}
Nov 03 20:14:09.387504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.381961    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:09.431085 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.429539    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"]
Nov 03 20:14:09.435563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.433457    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895" podStartSLOduration=2.433438772 podStartE2EDuration="2.433438772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:09.422278179 +0000 UTC m=+2201.577180598" watchObservedRunningTime="2026-11-03 20:14:09.433438772 +0000 UTC m=+2201.588341193"
Nov 03 20:14:09.439840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.439588    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.456513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.446994    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"]
Nov 03 20:14:09.519796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.519764    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"agc-cred-262850ebfa93\""
Nov 03 20:14:09.585695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.585657    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2cf6\" (UniqueName: \"kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-kube-api-access-j2cf6\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.585909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.585745    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.585909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.585857    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.586029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.585917    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.618362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.618323    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:09.688679 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.688571    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2cf6\" (UniqueName: \"kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-kube-api-access-j2cf6\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.688874 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.688691    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.688874 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.688756    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.688874 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.688806    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.690344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.690310    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.690500 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.690350    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.697216 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.694404    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.702218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.702193    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2cf6\" (UniqueName: \"kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-kube-api-access-j2cf6\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.765380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.765341    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:09.811892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.811857    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:09.897436 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.897399    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"]
Nov 03 20:14:09.902215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.902193    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:09.907305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.907277    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"payload-processing-external-model-e2e--e41c9c2e-dockercfg-mjgp2\""
Nov 03 20:14:09.907453 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.907387    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"openshift-service-ca.crt\""
Nov 03 20:14:09.907453 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.907444    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"e2e-cmp-legacy-946616-provider-cred\""
Nov 03 20:14:09.907648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.907534    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"kube-root-ca.crt\""
Nov 03 20:14:09.907648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.907590    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"routing-overlay\""
Nov 03 20:14:09.907757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.907662    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-eb69cb\""
Nov 03 20:14:09.908456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:09.908430    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"]
Nov 03 20:14:10.123010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.097045    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:10.123010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.097251    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:10.123010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.097852    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:10.123010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.097999    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbzh6\" (UniqueName: \"kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-kube-api-access-cbzh6\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:10.196279 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.196245    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"]
Nov 03 20:14:10.201125 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.200545    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:10.201410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.201389    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbzh6\" (UniqueName: \"kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-kube-api-access-cbzh6\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:10.205692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.205667    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:10.205990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.205962    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:10.210502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.207111    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:10.223668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.223637    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:10.251477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.243174    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:10.255278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.255244    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbzh6\" (UniqueName: \"kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-kube-api-access-cbzh6\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:10.391760 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.391681    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8" event={"ID":"760e2a75-4bdd-422d-879c-35d68e2eaa22","Type":"ContainerStarted","Data":"64dc9962e34cd5199006e45369dfb045e92c421f5b62b44dfc59f4ecf7da9288"}
Nov 03 20:14:10.391760 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.391741    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8" event={"ID":"760e2a75-4bdd-422d-879c-35d68e2eaa22","Type":"ContainerStarted","Data":"72157b219c10e7ca5d310189fb16e4f9aec7787a37a8fcc140af6f01b7175b63"}
Nov 03 20:14:10.412942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.412883    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8" podStartSLOduration=1.4128654680000001 podStartE2EDuration="1.412865468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:10.412393852 +0000 UTC m=+2202.567296273" watchObservedRunningTime="2026-11-03 20:14:10.412865468 +0000 UTC m=+2202.567767888"
Nov 03 20:14:10.521076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.521045    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:10.712628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.712545    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:10.816797 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:14:10.816748    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9133a5_8589_4761_8c80_4dd93a2195c6.slice/crio-c7922d50aecaed80a7b996f2f618f59297fda2032c839595cfeaafaee9f6e9be WatchSource:0}: Error finding container c7922d50aecaed80a7b996f2f618f59297fda2032c839595cfeaafaee9f6e9be: Status 404 returned error can't find the container with id c7922d50aecaed80a7b996f2f618f59297fda2032c839595cfeaafaee9f6e9be
Nov 03 20:14:10.817363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:10.817341    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"]
Nov 03 20:14:11.402468 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:11.402421    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6" event={"ID":"ad9133a5-8589-4761-8c80-4dd93a2195c6","Type":"ContainerStarted","Data":"5bcab7e7fe7f431316a1378cf9aae37c5dbcef4443736cad264be56098c19926"}
Nov 03 20:14:11.402909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:11.402545    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6" event={"ID":"ad9133a5-8589-4761-8c80-4dd93a2195c6","Type":"ContainerStarted","Data":"c7922d50aecaed80a7b996f2f618f59297fda2032c839595cfeaafaee9f6e9be"}
Nov 03 20:14:11.402909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:11.402719    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:11.402909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:11.402874    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:11.426158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:11.426109    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6" podStartSLOduration=2.426094673 podStartE2EDuration="2.426094673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:11.421231836 +0000 UTC m=+2203.576134323" watchObservedRunningTime="2026-11-03 20:14:11.426094673 +0000 UTC m=+2203.580997092"
Nov 03 20:14:11.438043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:11.438014    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:12.013598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:12.013562    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:12.138217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:12.138187    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:12.431533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:12.431399    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:12.635457 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:12.635409    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:13.318241 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.318209    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:13.320350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.320327    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"]
Nov 03 20:14:13.324525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.324478    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:13.332598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.332416    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"e2e-cmp-legacy-d84080-provider-cred\""
Nov 03 20:14:13.332598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.332433    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"openshift-service-ca.crt\""
Nov 03 20:14:13.332598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.332535    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec\""
Nov 03 20:14:13.332598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.332568    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"routing-overlay\""
Nov 03 20:14:13.332598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.332584    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"kube-root-ca.crt\""
Nov 03 20:14:13.335790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.335012    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"]
Nov 03 20:14:13.399609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.399570    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:13.399817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.399635    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:13.399817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.399797    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87clv\" (UniqueName: \"kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-kube-api-access-87clv\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:13.399929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.399892    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:13.501963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.501912    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:13.502403 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.502122    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:13.502403 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.502168    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:13.502403 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.502354    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87clv\" (UniqueName: \"kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-kube-api-access-87clv\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:13.502995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.502972    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:13.503206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.503177    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:13.504934 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.504911    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:13.550505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.547765    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87clv\" (UniqueName: \"kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-kube-api-access-87clv\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:13.644524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:13.644424    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:14.045150 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:14.045111    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:14.117901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:14.117858    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"]
Nov 03 20:14:14.118098 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:14:14.118062    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc97ad46_4774_4efa_a9fd_cb061eaea0d9.slice/crio-44c58b78bb16ca4f0a59f5a11589ec9a3da7e23d642bec41ebef55274b9eb7c6 WatchSource:0}: Error finding container 44c58b78bb16ca4f0a59f5a11589ec9a3da7e23d642bec41ebef55274b9eb7c6: Status 404 returned error can't find the container with id 44c58b78bb16ca4f0a59f5a11589ec9a3da7e23d642bec41ebef55274b9eb7c6
Nov 03 20:14:14.427908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:14.427819    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr" event={"ID":"dc97ad46-4774-4efa-a9fd-cb061eaea0d9","Type":"ContainerStarted","Data":"0acd073c614eb322c4ee43b7063e24fe4fbba35d093769519ef64d2f27c13c7b"}
Nov 03 20:14:14.427908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:14.427885    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:14.427908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:14.427906    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr" event={"ID":"dc97ad46-4774-4efa-a9fd-cb061eaea0d9","Type":"ContainerStarted","Data":"44c58b78bb16ca4f0a59f5a11589ec9a3da7e23d642bec41ebef55274b9eb7c6"}
Nov 03 20:14:14.450495 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:14.450432    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr" podStartSLOduration=1.450413166 podStartE2EDuration="1.450413166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:14.446950189 +0000 UTC m=+2206.601852609" watchObservedRunningTime="2026-11-03 20:14:14.450413166 +0000 UTC m=+2206.605315586"
Nov 03 20:14:14.789964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:14.789927    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:15.219162 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:14:15.219121    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:14:15.546996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:15.546909    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:16.248305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:16.248269    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:16.613598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:16.613507    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"]
Nov 03 20:14:16.613809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:16.613775    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg" podUID="4d55596f-ce16-4eab-9c20-19ea62660132" containerName="istio-proxy" containerID="cri-o://8d61a0fce85263f9c4514723aa08f1ebbdd38137eee7b7ab08e1cb7b39348ca0" gracePeriod=30
Nov 03 20:14:16.960588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:16.957420    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:17.638385 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:17.638348    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:18.363938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:18.363905    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:19.042853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:19.042806    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:19.743646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:19.743609    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:20.395875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:20.395841    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:20.450393 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:20.450337    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:21.128786 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:21.128747    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:21.913030 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:21.913006    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 20:14:21.914205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:21.914180    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:22.021225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.020441    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-socket\") pod \"4d55596f-ce16-4eab-9c20-19ea62660132\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") "
Nov 03 20:14:22.021225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.020569    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-data\") pod \"4d55596f-ce16-4eab-9c20-19ea62660132\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") "
Nov 03 20:14:22.021225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.020630    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-certs\") pod \"4d55596f-ce16-4eab-9c20-19ea62660132\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") "
Nov 03 20:14:22.021225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.020660    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d55596f-ce16-4eab-9c20-19ea62660132-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d55596f-ce16-4eab-9c20-19ea62660132-istio-podinfo\") pod \"4d55596f-ce16-4eab-9c20-19ea62660132\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") "
Nov 03 20:14:22.021225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.020732    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-credential-socket\") pod \"4d55596f-ce16-4eab-9c20-19ea62660132\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") "
Nov 03 20:14:22.021225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.020751    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d55596f-ce16-4eab-9c20-19ea62660132-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d55596f-ce16-4eab-9c20-19ea62660132-istiod-ca-cert\") pod \"4d55596f-ce16-4eab-9c20-19ea62660132\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") "
Nov 03 20:14:22.021225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.020779    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-envoy\") pod \"4d55596f-ce16-4eab-9c20-19ea62660132\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") "
Nov 03 20:14:22.021225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.020831    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-kube-api-access-wzdrr\" (UniqueName: \"kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-kube-api-access-wzdrr\") pod \"4d55596f-ce16-4eab-9c20-19ea62660132\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") "
Nov 03 20:14:22.021225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.020858    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-socket" pod "4d55596f-ce16-4eab-9c20-19ea62660132" (UID: "4d55596f-ce16-4eab-9c20-19ea62660132"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:22.021225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.020883    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-istio-token\" (UniqueName: \"kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-istio-token\") pod \"4d55596f-ce16-4eab-9c20-19ea62660132\" (UID: \"4d55596f-ce16-4eab-9c20-19ea62660132\") "
Nov 03 20:14:22.022064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.021639    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-credential-socket" pod "4d55596f-ce16-4eab-9c20-19ea62660132" (UID: "4d55596f-ce16-4eab-9c20-19ea62660132"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:22.022064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.021861    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-data" pod "4d55596f-ce16-4eab-9c20-19ea62660132" (UID: "4d55596f-ce16-4eab-9c20-19ea62660132"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:22.022064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.021892    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d55596f-ce16-4eab-9c20-19ea62660132-istiod-ca-cert" pod "4d55596f-ce16-4eab-9c20-19ea62660132" (UID: "4d55596f-ce16-4eab-9c20-19ea62660132"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:14:22.022233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.022095    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-certs" pod "4d55596f-ce16-4eab-9c20-19ea62660132" (UID: "4d55596f-ce16-4eab-9c20-19ea62660132"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:22.022233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.022130    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:22.022233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.022153    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d55596f-ce16-4eab-9c20-19ea62660132-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:22.022233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.022169    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:22.022233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.022202    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:22.026983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.026921    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-envoy" pod "4d55596f-ce16-4eab-9c20-19ea62660132" (UID: "4d55596f-ce16-4eab-9c20-19ea62660132"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:22.028434 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.028405    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-istio-token" pod "4d55596f-ce16-4eab-9c20-19ea62660132" (UID: "4d55596f-ce16-4eab-9c20-19ea62660132"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:22.028622 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.028602    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d55596f-ce16-4eab-9c20-19ea62660132-istio-podinfo" pod "4d55596f-ce16-4eab-9c20-19ea62660132" (UID: "4d55596f-ce16-4eab-9c20-19ea62660132"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:14:22.028779 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.028755    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-kube-api-access-wzdrr" pod "4d55596f-ce16-4eab-9c20-19ea62660132" (UID: "4d55596f-ce16-4eab-9c20-19ea62660132"). InnerVolumeSpecName "kube-api-access-wzdrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:22.124039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.124005    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzdrr\" (UniqueName: \"kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-kube-api-access-wzdrr\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:22.124039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.124036    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d55596f-ce16-4eab-9c20-19ea62660132-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:22.124039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.124046    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:22.124291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.124055    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d55596f-ce16-4eab-9c20-19ea62660132-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:22.124291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.124066    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d55596f-ce16-4eab-9c20-19ea62660132-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:22.413526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.413475    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:14:22.413791 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.413771    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:14:22.486469 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.486278    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"]
Nov 03 20:14:22.523137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.523093    5659 generic.go:356] "Generic (PLEG): container finished" podID="4d55596f-ce16-4eab-9c20-19ea62660132" containerID="8d61a0fce85263f9c4514723aa08f1ebbdd38137eee7b7ab08e1cb7b39348ca0" exitCode=0
Nov 03 20:14:22.523137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.523139    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"
Nov 03 20:14:22.523414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.523174    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg" event={"ID":"4d55596f-ce16-4eab-9c20-19ea62660132","Type":"ContainerDied","Data":"8d61a0fce85263f9c4514723aa08f1ebbdd38137eee7b7ab08e1cb7b39348ca0"}
Nov 03 20:14:22.523414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.523243    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg" event={"ID":"4d55596f-ce16-4eab-9c20-19ea62660132","Type":"ContainerDied","Data":"819de2c6ce6df617730d3258ad19c1e921fc0ebebccd78933301cd91e55fac62"}
Nov 03 20:14:22.523414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.523277    5659 scope.go:122] "RemoveContainer" containerID="8d61a0fce85263f9c4514723aa08f1ebbdd38137eee7b7ab08e1cb7b39348ca0"
Nov 03 20:14:22.542910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.537695    5659 scope.go:122] "RemoveContainer" containerID="8d61a0fce85263f9c4514723aa08f1ebbdd38137eee7b7ab08e1cb7b39348ca0"
Nov 03 20:14:22.542910 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:14:22.538106    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d61a0fce85263f9c4514723aa08f1ebbdd38137eee7b7ab08e1cb7b39348ca0\": container with ID starting with 8d61a0fce85263f9c4514723aa08f1ebbdd38137eee7b7ab08e1cb7b39348ca0 not found: ID does not exist" containerID="8d61a0fce85263f9c4514723aa08f1ebbdd38137eee7b7ab08e1cb7b39348ca0"
Nov 03 20:14:22.542910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.538146    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d61a0fce85263f9c4514723aa08f1ebbdd38137eee7b7ab08e1cb7b39348ca0"} err="failed to get container status \"8d61a0fce85263f9c4514723aa08f1ebbdd38137eee7b7ab08e1cb7b39348ca0\": rpc error: code = NotFound desc = could not find container \"8d61a0fce85263f9c4514723aa08f1ebbdd38137eee7b7ab08e1cb7b39348ca0\": container with ID starting with 8d61a0fce85263f9c4514723aa08f1ebbdd38137eee7b7ab08e1cb7b39348ca0 not found: ID does not exist"
Nov 03 20:14:22.555603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.555569    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"]
Nov 03 20:14:22.559137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.559107    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-lr9tg"]
Nov 03 20:14:22.821369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:22.821333    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:23.594628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:23.590351    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:23.986321 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:23.986285    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d55596f-ce16-4eab-9c20-19ea62660132" path="/var/lib/kubelet/pods/4d55596f-ce16-4eab-9c20-19ea62660132/volumes"
Nov 03 20:14:24.341196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:24.341096    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:25.131569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:25.131521    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:25.445304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:25.445206    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:14:25.845362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:25.845315    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:26.548396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:26.548360    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:27.324099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.324061    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:27.487240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.487191    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895" podUID="61b57093-ed30-4994-bfc5-2f6901e467b9" containerName="payload-processing" containerID="cri-o://045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513" gracePeriod=25
Nov 03 20:14:27.800740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.800713    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:27.835401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.835363    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-plugins-config-volume\") pod \"61b57093-ed30-4994-bfc5-2f6901e467b9\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") "
Nov 03 20:14:27.835687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.835452    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-provider-credentials\") pod \"61b57093-ed30-4994-bfc5-2f6901e467b9\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") "
Nov 03 20:14:27.835795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.835771    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-routing-overlay-volume\") pod \"61b57093-ed30-4994-bfc5-2f6901e467b9\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") "
Nov 03 20:14:27.835926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.835902    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-plugins-config-volume" pod "61b57093-ed30-4994-bfc5-2f6901e467b9" (UID: "61b57093-ed30-4994-bfc5-2f6901e467b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:14:27.836098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.836076    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-routing-overlay-volume" pod "61b57093-ed30-4994-bfc5-2f6901e467b9" (UID: "61b57093-ed30-4994-bfc5-2f6901e467b9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:14:27.836290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.836191    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-kube-api-access-8ttvk\" (UniqueName: \"kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-kube-api-access-8ttvk\") pod \"61b57093-ed30-4994-bfc5-2f6901e467b9\" (UID: \"61b57093-ed30-4994-bfc5-2f6901e467b9\") "
Nov 03 20:14:27.837154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.837130    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.837268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.837156    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61b57093-ed30-4994-bfc5-2f6901e467b9-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.839298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.839275    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-kube-api-access-8ttvk" pod "61b57093-ed30-4994-bfc5-2f6901e467b9" (UID: "61b57093-ed30-4994-bfc5-2f6901e467b9"). InnerVolumeSpecName "kube-api-access-8ttvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:27.862057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.862017    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-provider-credentials" pod "61b57093-ed30-4994-bfc5-2f6901e467b9" (UID: "61b57093-ed30-4994-bfc5-2f6901e467b9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:27.938460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.938424    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ttvk\" (UniqueName: \"kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-kube-api-access-8ttvk\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.938460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.938456    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61b57093-ed30-4994-bfc5-2f6901e467b9-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.986654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:27.986620    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:28.141101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.140962    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"]
Nov 03 20:14:28.142612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.142581    5659 cpu_manager.go:414] "containerMap: removing container" podUID="61b57093-ed30-4994-bfc5-2f6901e467b9" containerName="payload-processing"
Nov 03 20:14:28.142612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.142610    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b57093-ed30-4994-bfc5-2f6901e467b9" containerName="payload-processing"
Nov 03 20:14:28.142796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.142666    5659 cpu_manager.go:414] "containerMap: removing container" podUID="4d55596f-ce16-4eab-9c20-19ea62660132" containerName="istio-proxy"
Nov 03 20:14:28.142796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.142674    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d55596f-ce16-4eab-9c20-19ea62660132" containerName="istio-proxy"
Nov 03 20:14:28.142935 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.142917    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b57093-ed30-4994-bfc5-2f6901e467b9" containerName="payload-processing"
Nov 03 20:14:28.142995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.142946    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d55596f-ce16-4eab-9c20-19ea62660132" containerName="istio-proxy"
Nov 03 20:14:28.157537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.148035    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.187438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.187407    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"]
Nov 03 20:14:28.240282 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.240245    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-eb69cb-openshift-default-dockercfg-w96q5\""
Nov 03 20:14:28.255406 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.255350    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.255611 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.255463    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.255659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.255619    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.255792 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.255767    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.255971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.255854    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22zkb\" (UniqueName: \"kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-kube-api-access-22zkb\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.256072 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.256048    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.256165 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.256152    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4750b2-2d03-49c3-ab74-07df22747954-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.256227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.256183    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4750b2-2d03-49c3-ab74-07df22747954-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.256302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.256285    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.358118 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.358072    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.358303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.358165    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22zkb\" (UniqueName: \"kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-kube-api-access-22zkb\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.358356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.358336    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.358459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.358443    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4750b2-2d03-49c3-ab74-07df22747954-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.358539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.358505    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4750b2-2d03-49c3-ab74-07df22747954-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.358602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.358547    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.358602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.358582    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.358706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.358622    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.358706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.358700    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.358836 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.358814    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.358948 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.358922    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.359228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.359193    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.359368 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.359225    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.359449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.359426    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4750b2-2d03-49c3-ab74-07df22747954-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.361583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.361559    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.361831 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.361813    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4750b2-2d03-49c3-ab74-07df22747954-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.370272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.370237    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.370516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.370463    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22zkb\" (UniqueName: \"kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-kube-api-access-22zkb\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.500300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.500267    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:28.606901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.605184    5659 generic.go:356] "Generic (PLEG): container finished" podID="61b57093-ed30-4994-bfc5-2f6901e467b9" containerID="045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513" exitCode=0
Nov 03 20:14:28.606901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.605363    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895" event={"ID":"61b57093-ed30-4994-bfc5-2f6901e467b9","Type":"ContainerDied","Data":"045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513"}
Nov 03 20:14:28.606901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.605430    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895" event={"ID":"61b57093-ed30-4994-bfc5-2f6901e467b9","Type":"ContainerDied","Data":"51a1dfce39cdfd3ea10cb839955e1c4d02164a3580f3e2326df7672e8aa0f402"}
Nov 03 20:14:28.606901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.605452    5659 scope.go:122] "RemoveContainer" containerID="045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513"
Nov 03 20:14:28.606901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.605605    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"
Nov 03 20:14:28.620176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.620067    5659 scope.go:122] "RemoveContainer" containerID="045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513"
Nov 03 20:14:28.620556 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:14:28.620466    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513\": container with ID starting with 045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513 not found: ID does not exist" containerID="045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513"
Nov 03 20:14:28.620556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.620526    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513"} err="failed to get container status \"045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513\": rpc error: code = NotFound desc = could not find container \"045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513\": container with ID starting with 045c58a815abda8606231b8089ded3df77955ff82e84d947e6b34b7be06ec513 not found: ID does not exist"
Nov 03 20:14:28.628213 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.628182    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"]
Nov 03 20:14:28.632932 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.632903    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7c5cmz895"]
Nov 03 20:14:28.795451 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:14:28.795385    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4750b2_2d03_49c3_ab74_07df22747954.slice/crio-3f9e10f49f15280c5bac537f517e1dd636b529eca2b41699910b11e413ed8b16 WatchSource:0}: Error finding container 3f9e10f49f15280c5bac537f517e1dd636b529eca2b41699910b11e413ed8b16: Status 404 returned error can't find the container with id 3f9e10f49f15280c5bac537f517e1dd636b529eca2b41699910b11e413ed8b16
Nov 03 20:14:28.796785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.796761    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"]
Nov 03 20:14:28.798788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.798751    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:28.798911 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.798821    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:28.798911 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.798857    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:28.845594 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:28.845564    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:29.616867 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:29.616827    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj" event={"ID":"1a4750b2-2d03-49c3-ab74-07df22747954","Type":"ContainerStarted","Data":"0e982b0fb27c6895de2690425cb489542eb08f2ce6ae2e8d0ebb5739b1cd6201"}
Nov 03 20:14:29.617092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:29.616888    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj" event={"ID":"1a4750b2-2d03-49c3-ab74-07df22747954","Type":"ContainerStarted","Data":"3f9e10f49f15280c5bac537f517e1dd636b529eca2b41699910b11e413ed8b16"}
Nov 03 20:14:29.672008 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:29.671735    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj" podStartSLOduration=1.671718547 podStartE2EDuration="1.671718547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:29.642194964 +0000 UTC m=+2221.797097414" watchObservedRunningTime="2026-11-03 20:14:29.671718547 +0000 UTC m=+2221.826620966"
Nov 03 20:14:29.782912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:29.782874    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:29.980559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:29.980524    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b57093-ed30-4994-bfc5-2f6901e467b9" path="/var/lib/kubelet/pods/61b57093-ed30-4994-bfc5-2f6901e467b9/volumes"
Nov 03 20:14:30.500897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:30.500864    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:30.502352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:30.502322    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 03 20:14:30.502506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:30.502376    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj" podUID="1a4750b2-2d03-49c3-ab74-07df22747954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 03 20:14:30.685363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:30.679613    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:31.501505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:31.501120    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 03 20:14:31.501505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:31.501275    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj" podUID="1a4750b2-2d03-49c3-ab74-07df22747954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 03 20:14:32.417761 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:32.417717    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:32.504161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:32.504116    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 03 20:14:32.504844 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:32.504786    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj" podUID="1a4750b2-2d03-49c3-ab74-07df22747954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 03 20:14:33.330509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:33.323496    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:33.501419 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:33.501382    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 03 20:14:33.501649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:33.501453    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj" podUID="1a4750b2-2d03-49c3-ab74-07df22747954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 03 20:14:34.097530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:34.095825    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:34.501670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:34.501631    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 03 20:14:34.501857 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:34.501687    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj" podUID="1a4750b2-2d03-49c3-ab74-07df22747954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 03 20:14:35.002726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:35.002689    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:35.500842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:35.500803    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 03 20:14:35.501278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:35.500871    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj" podUID="1a4750b2-2d03-49c3-ab74-07df22747954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 03 20:14:35.763566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:35.763477    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:36.506113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:36.506079    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:36.506576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:36.506374    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:36.507141 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:36.507122    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:14:36.557195 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:36.557156    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:37.356721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:37.356689    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:38.033057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:38.033022    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:38.616017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:38.615980    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:38.948200 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:38.948109    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:39.639405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:39.639369    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:40.333043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:40.332961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:41.100061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:41.100027    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:41.819568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:41.819533    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:42.460111 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:42.460075    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:43.156353 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:43.156309    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:44.004237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:43.978762    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:44.658303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:44.658270    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:45.218909 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:14:45.218871    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:14:45.426635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:45.426608    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:46.113001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:46.112970    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:46.790515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:46.790453    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:47.493082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:47.493049    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:48.154687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:48.154646    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:48.831264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:48.831234    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:49.606225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:49.606191    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:50.344856 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:50.344773    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:51.030354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:51.030320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:51.795007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:51.794968    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:52.885903 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:52.885873    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:53.756986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:53.756897    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:54.435216 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:54.435175    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:54.565257 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:54.565223    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:55.739965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:55.739926    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:56.529395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:56.529362    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:56.770404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:56.770360    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:56.921571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:56.921173    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:57.730810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:57.730775    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:58.188539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.188423    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"]
Nov 03 20:14:58.194594 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.194564    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.195228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.195199    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"]
Nov 03 20:14:58.200381 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.200348    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.217810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.217776    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"]
Nov 03 20:14:58.229591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.229558    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"]
Nov 03 20:14:58.232961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.232937    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"]
Nov 03 20:14:58.245898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.245864    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.249858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.249826    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"]
Nov 03 20:14:58.258858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.258826    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwkc2\" (UniqueName: \"kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-kube-api-access-gwkc2\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.259016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.258932    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.259016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.258993    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.259142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259036    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.259142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259069    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc874822-913a-4d2d-922f-19d6095c34f9-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.259142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259095    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.259142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259118    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.259378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259148    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15cce432-d79b-4f87-b5c6-63908192552d-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.259378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259174    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.259378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259206    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.259378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259232    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzghp\" (UniqueName: \"kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-kube-api-access-zzghp\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.259378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259273    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.259752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259435    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc874822-913a-4d2d-922f-19d6095c34f9-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.259752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259549    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.259752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259665    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.259752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259697    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.260372 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259763    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.260372 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.259812    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15cce432-d79b-4f87-b5c6-63908192552d-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.362043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362000    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.362240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362061    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.362240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362215    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15cce432-d79b-4f87-b5c6-63908192552d-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.362334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362315    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwkc2\" (UniqueName: \"kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-kube-api-access-gwkc2\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.362384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362366    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.362447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362415    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.362519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362451    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.362519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362508    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.362625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362545    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc874822-913a-4d2d-922f-19d6095c34f9-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.362625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362575    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.362625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362601    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.362789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362638    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15cce432-d79b-4f87-b5c6-63908192552d-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.362789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362669    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.362789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362703    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.362789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362729    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzghp\" (UniqueName: \"kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-kube-api-access-zzghp\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.362789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362776    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.363071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362794    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.363071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362892    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992549e4-0f0c-4a0e-a170-a472285aa24f-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.363071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.362974    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc874822-913a-4d2d-922f-19d6095c34f9-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.363071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363002    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.363300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363070    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.363300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363112    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.363300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363162    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.363300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363195    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.363300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363242    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcpm5\" (UniqueName: \"kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-kube-api-access-xcpm5\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.363300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363280    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.363657 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363396    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc874822-913a-4d2d-922f-19d6095c34f9-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.363657 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363630    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.363776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363736    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.363828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363784    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.363888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363826    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.363888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363864    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.363888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.363862    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.364267 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.364245    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.365437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.364559    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.365437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.364711    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15cce432-d79b-4f87-b5c6-63908192552d-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.365437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.364923    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.365437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.364946    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.365437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.365268    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15cce432-d79b-4f87-b5c6-63908192552d-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.366673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.366648    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.367025 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.367008    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc874822-913a-4d2d-922f-19d6095c34f9-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.375223 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.375195    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.375432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.375407    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwkc2\" (UniqueName: \"kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-kube-api-access-gwkc2\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.376157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.376130    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzghp\" (UniqueName: \"kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-kube-api-access-zzghp\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.376380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.376358    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.465711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.465671    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.465929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.465757    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.465929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.465894    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.466080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.466064    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992549e4-0f0c-4a0e-a170-a472285aa24f-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.466147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.466113    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.466147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.466137    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.466261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.466160    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.466261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.466195    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.466261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.466220    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.466261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.466231    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.466511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.466278    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.466511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.466320    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcpm5\" (UniqueName: \"kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-kube-api-access-xcpm5\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.466782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.466758    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992549e4-0f0c-4a0e-a170-a472285aa24f-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.466853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.466824    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.468775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.468748    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.468903 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.468750    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.478920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.478893    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcpm5\" (UniqueName: \"kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-kube-api-access-xcpm5\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.479303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.479283    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.523880 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.523846    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:58.533467 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.533442    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:58.544721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.543758    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:58.574517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.569502    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:14:58.911796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.911759    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"]
Nov 03 20:14:58.915555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.915509    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:58.915678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.915599    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:58.915678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.915642    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:58.916907 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:14:58.916884    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc874822_913a_4d2d_922f_19d6095c34f9.slice/crio-71828ff694fba8f6a54253e82abcc0c8ad7dbc7a6ccbb112a1048c5a590cfaaa WatchSource:0}: Error finding container 71828ff694fba8f6a54253e82abcc0c8ad7dbc7a6ccbb112a1048c5a590cfaaa: Status 404 returned error can't find the container with id 71828ff694fba8f6a54253e82abcc0c8ad7dbc7a6ccbb112a1048c5a590cfaaa
Nov 03 20:14:58.918244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.918220    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"]
Nov 03 20:14:58.919771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.919743    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:58.919856 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.919814    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:58.919922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.919863    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:58.945896 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.945866    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"]
Nov 03 20:14:58.964816 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:14:58.964778    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992549e4_0f0c_4a0e_a170_a472285aa24f.slice/crio-050a9a8dfde5045e859a7afb1e8996c7dbd81f9eb0317c910711a22639401771 WatchSource:0}: Error finding container 050a9a8dfde5045e859a7afb1e8996c7dbd81f9eb0317c910711a22639401771: Status 404 returned error can't find the container with id 050a9a8dfde5045e859a7afb1e8996c7dbd81f9eb0317c910711a22639401771
Nov 03 20:14:58.967342 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.967268    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:58.967534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.967368    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:58.967534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:58.967421    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:59.221042 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:59.221006    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" event={"ID":"15cce432-d79b-4f87-b5c6-63908192552d","Type":"ContainerStarted","Data":"3c2ea4341de31f162e7ba96742d68a316e58807241967c5b3bda4b3c7d480044"}
Nov 03 20:14:59.223772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:59.223745    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" event={"ID":"15cce432-d79b-4f87-b5c6-63908192552d","Type":"ContainerStarted","Data":"fa21a220c37bc8e9e7466e33ca54a91303c5d26d9222d9401e324b4272678384"}
Nov 03 20:14:59.268947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:59.267202    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" event={"ID":"cc874822-913a-4d2d-922f-19d6095c34f9","Type":"ContainerStarted","Data":"393c2290a69b81efe255c23e740d2763fbc15702f66320dbb1cf9a90d6a5f860"}
Nov 03 20:14:59.268947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:59.267274    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" event={"ID":"cc874822-913a-4d2d-922f-19d6095c34f9","Type":"ContainerStarted","Data":"71828ff694fba8f6a54253e82abcc0c8ad7dbc7a6ccbb112a1048c5a590cfaaa"}
Nov 03 20:14:59.270905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:59.270844    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:59.277405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:59.275530    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" event={"ID":"992549e4-0f0c-4a0e-a170-a472285aa24f","Type":"ContainerStarted","Data":"0562e2f13fc22b13548d99578d92dbe854d9be4587231233e9183fab77224c35"}
Nov 03 20:14:59.277405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:59.275599    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" event={"ID":"992549e4-0f0c-4a0e-a170-a472285aa24f","Type":"ContainerStarted","Data":"050a9a8dfde5045e859a7afb1e8996c7dbd81f9eb0317c910711a22639401771"}
Nov 03 20:14:59.316393 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:59.316334    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" podStartSLOduration=1.316314793 podStartE2EDuration="1.316314793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:59.304042342 +0000 UTC m=+2251.458944773" watchObservedRunningTime="2026-11-03 20:14:59.316314793 +0000 UTC m=+2251.471217215"
Nov 03 20:14:59.326335 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:59.323361    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" podStartSLOduration=1.323342282 podStartE2EDuration="1.323342282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:59.26865525 +0000 UTC m=+2251.423557669" watchObservedRunningTime="2026-11-03 20:14:59.323342282 +0000 UTC m=+2251.478244706"
Nov 03 20:14:59.344251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:59.344170    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" podStartSLOduration=1.344146098 podStartE2EDuration="1.344146098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:59.330954715 +0000 UTC m=+2251.485857134" watchObservedRunningTime="2026-11-03 20:14:59.344146098 +0000 UTC m=+2251.499048519"
Nov 03 20:14:59.524715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:59.524678    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:14:59.534580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:59.534457    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:14:59.574552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:14:59.573842    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:15:00.256781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.256743    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"]
Nov 03 20:15:00.368389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.366092    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:00.438547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.375760    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"
Nov 03 20:15:00.438547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.393435    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\""
Nov 03 20:15:00.529470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.528645    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 20:15:00.529470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.528705    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" podUID="cc874822-913a-4d2d-922f-19d6095c34f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 20:15:00.614029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.604504    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180b9b66-9700-4b5e-bb2b-fb979d97a34b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84\" (UID: \"180b9b66-9700-4b5e-bb2b-fb979d97a34b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"
Nov 03 20:15:00.614029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.604603    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvd2f\" (UniqueName: \"kubernetes.io/projected/180b9b66-9700-4b5e-bb2b-fb979d97a34b-kube-api-access-mvd2f\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84\" (UID: \"180b9b66-9700-4b5e-bb2b-fb979d97a34b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"
Nov 03 20:15:00.627508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.618638    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 20:15:00.627508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.618714    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" podUID="15cce432-d79b-4f87-b5c6-63908192552d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 20:15:00.663505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.662530    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"]
Nov 03 20:15:00.732510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.732056    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:15:00.732510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.732135    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" podUID="992549e4-0f0c-4a0e-a170-a472285aa24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 20:15:00.757515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.757017    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180b9b66-9700-4b5e-bb2b-fb979d97a34b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84\" (UID: \"180b9b66-9700-4b5e-bb2b-fb979d97a34b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"
Nov 03 20:15:00.757515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.757118    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvd2f\" (UniqueName: \"kubernetes.io/projected/180b9b66-9700-4b5e-bb2b-fb979d97a34b-kube-api-access-mvd2f\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84\" (UID: \"180b9b66-9700-4b5e-bb2b-fb979d97a34b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"
Nov 03 20:15:00.782566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.759506    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180b9b66-9700-4b5e-bb2b-fb979d97a34b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84\" (UID: \"180b9b66-9700-4b5e-bb2b-fb979d97a34b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"
Nov 03 20:15:00.821904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.809590    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:00.833642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.826456    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"
Nov 03 20:15:00.906512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.891174    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvd2f\" (UniqueName: \"kubernetes.io/projected/180b9b66-9700-4b5e-bb2b-fb979d97a34b-kube-api-access-mvd2f\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84\" (UID: \"180b9b66-9700-4b5e-bb2b-fb979d97a34b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"
Nov 03 20:15:00.906512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.896970    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx5ct\" (UniqueName: \"kubernetes.io/projected/d5f116fd-1e4c-4b58-b070-6c48e132562c-kube-api-access-zx5ct\") pod \"maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk\" (UID: \"d5f116fd-1e4c-4b58-b070-6c48e132562c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"
Nov 03 20:15:00.906512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.897032    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f116fd-1e4c-4b58-b070-6c48e132562c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk\" (UID: \"d5f116fd-1e4c-4b58-b070-6c48e132562c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"
Nov 03 20:15:00.950580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:00.950538    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"]
Nov 03 20:15:01.030520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.028622    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx5ct\" (UniqueName: \"kubernetes.io/projected/d5f116fd-1e4c-4b58-b070-6c48e132562c-kube-api-access-zx5ct\") pod \"maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk\" (UID: \"d5f116fd-1e4c-4b58-b070-6c48e132562c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"
Nov 03 20:15:01.030520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.028783    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f116fd-1e4c-4b58-b070-6c48e132562c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk\" (UID: \"d5f116fd-1e4c-4b58-b070-6c48e132562c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"
Nov 03 20:15:01.078564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.069167    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f116fd-1e4c-4b58-b070-6c48e132562c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk\" (UID: \"d5f116fd-1e4c-4b58-b070-6c48e132562c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"
Nov 03 20:15:01.125003 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.124894    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"]
Nov 03 20:15:01.169525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.169249    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"
Nov 03 20:15:01.264506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.241296    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx5ct\" (UniqueName: \"kubernetes.io/projected/d5f116fd-1e4c-4b58-b070-6c48e132562c-kube-api-access-zx5ct\") pod \"maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk\" (UID: \"d5f116fd-1e4c-4b58-b070-6c48e132562c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"
Nov 03 20:15:01.348564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.347495    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:01.401507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.394032    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:15:01.401507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.395454    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:15:01.426508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.423589    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:01.507570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.501036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:01.526105 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.526067    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 20:15:01.526443 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.526395    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" podUID="cc874822-913a-4d2d-922f-19d6095c34f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 20:15:01.540296 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.539761    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:01.599810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.540653    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 20:15:01.599810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.540722    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" podUID="15cce432-d79b-4f87-b5c6-63908192552d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 20:15:01.599810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.551019    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"
Nov 03 20:15:01.599810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.557193    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:15:01.632505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.613880    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:15:01.632505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.613942    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" podUID="992549e4-0f0c-4a0e-a170-a472285aa24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 20:15:02.052605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:01.929652    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:02.416508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:02.408179    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:15:02.550512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:02.541423    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:02.661127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:02.641191    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:02.664776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:02.651237    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:02.694514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:02.676476    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:02.694514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:02.653048    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 20:15:02.694514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:02.676829    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" podUID="cc874822-913a-4d2d-922f-19d6095c34f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 20:15:02.694514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:02.653414    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 20:15:02.694514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:02.676893    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" podUID="15cce432-d79b-4f87-b5c6-63908192552d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 20:15:02.695208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:02.640478    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:02.754412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:02.752660    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:15:02.754412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:02.752732    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" podUID="992549e4-0f0c-4a0e-a170-a472285aa24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 20:15:03.193562 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:03.183861    5659 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.215s"
Nov 03 20:15:03.193562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.183947    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:15:03.193562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.183980    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"]
Nov 03 20:15:03.193562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.184043    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:15:03.193562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.184133    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:15:03.193562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.184163    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:15:03.193562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.184197    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 20:15:03.193562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.184234    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 20:15:03.365530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.356631    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:03.646466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.646423    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:15:03.655456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.655415    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" podUID="992549e4-0f0c-4a0e-a170-a472285aa24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 20:15:03.669304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.669257    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:03.684596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.680042    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 20:15:03.684596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.680101    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" podUID="cc874822-913a-4d2d-922f-19d6095c34f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 20:15:03.826026 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.825978    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:03.902265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.826311    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:03.917864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.873135    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 20:15:03.917864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.914212    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" podUID="15cce432-d79b-4f87-b5c6-63908192552d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 20:15:03.917864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.894200    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:04.011506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:03.997596    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:04.112721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.112605    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Nov 03 20:15:04.306126 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:15:04.306090    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f116fd_1e4c_4b58_b070_6c48e132562c.slice/crio-b9a5ee9fbc405c640a0f29acf4379f2795bbcb70e6d5237611773d7f8359b964 WatchSource:0}: Error finding container b9a5ee9fbc405c640a0f29acf4379f2795bbcb70e6d5237611773d7f8359b964: Status 404 returned error can't find the container with id b9a5ee9fbc405c640a0f29acf4379f2795bbcb70e6d5237611773d7f8359b964
Nov 03 20:15:04.419009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.411898    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:15:04.547525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.536155    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 20:15:04.547525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.536220    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" podUID="cc874822-913a-4d2d-922f-19d6095c34f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 20:15:04.549413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.548171    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:04.561612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.559445    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 20:15:04.561612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.560013    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" podUID="15cce432-d79b-4f87-b5c6-63908192552d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 20:15:04.585660 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.585625    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:15:04.585952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.585923    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" podUID="992549e4-0f0c-4a0e-a170-a472285aa24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 20:15:04.790748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.778933    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:04.904979 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.904935    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:15:04.992513 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:04.992303    5659 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.02s"
Nov 03 20:15:04.992513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.992380    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84" event={"ID":"180b9b66-9700-4b5e-bb2b-fb979d97a34b","Type":"ContainerStarted","Data":"7989e1a08e6fe79887ad5605602ac91aedb3ba80c863c9f1fe3e78632893943a"}
Nov 03 20:15:04.992759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.992582    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"]
Nov 03 20:15:05.008547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.996641    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 20:15:05.008547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.996736    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 20:15:05.008547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:04.996798    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:15:05.008547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.001319    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:05.059632 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.031446    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 20:15:05.615643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.592801    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 20:15:05.615643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.592889    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" podUID="cc874822-913a-4d2d-922f-19d6095c34f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 20:15:05.615643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.602184    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:05.665600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.658426    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:15:05.665600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.658501    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" podUID="992549e4-0f0c-4a0e-a170-a472285aa24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 20:15:05.675898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.675841    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 03 20:15:05.690648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.690599    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:05.711948 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.711848    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:05.803504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.802790    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 20:15:05.813725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.813671    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 20:15:05.814006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.813961    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" podUID="15cce432-d79b-4f87-b5c6-63908192552d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 20:15:05.921920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.921658    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:15:05.939680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:05.939623    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:15:06.024446 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:06.024150    5659 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180b9b66_9700_4b5e_bb2b_fb979d97a34b.slice/crio-8c9d4e8d8fdfd0a8aab7574f648085d647817e7a90214cc0a30b7c14c47d3657.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:15:06.095119 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:06.095076    5659 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180b9b66_9700_4b5e_bb2b_fb979d97a34b.slice/crio-conmon-8c9d4e8d8fdfd0a8aab7574f648085d647817e7a90214cc0a30b7c14c47d3657.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:15:06.636565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:06.631422    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:06.753509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:06.752654    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 20:15:06.753509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:06.752743    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" podUID="cc874822-913a-4d2d-922f-19d6095c34f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 20:15:06.856507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:06.807417    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:06.903568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:06.890427    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:15:06.903568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:06.890539    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" podUID="992549e4-0f0c-4a0e-a170-a472285aa24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 20:15:06.903568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:06.890659    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 20:15:06.903568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:06.890693    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" podUID="15cce432-d79b-4f87-b5c6-63908192552d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 20:15:06.962510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:06.937637    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:15:06.962510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:06.943631    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:06.992138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:06.992073    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:15:07.044028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.043634    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:07.328395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.328312    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84" podStartSLOduration=7.328285287 podStartE2EDuration="7.328285287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:06.855939468 +0000 UTC m=+2259.010841880" watchObservedRunningTime="2026-11-03 20:15:07.328285287 +0000 UTC m=+2259.483187705"
Nov 03 20:15:07.332119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.332084    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:07.414678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.414120    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 20:15:07.427806 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:07.427768    5659 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.454s"
Nov 03 20:15:07.428121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.428086    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84" event={"ID":"180b9b66-9700-4b5e-bb2b-fb979d97a34b","Type":"ContainerStarted","Data":"8c9d4e8d8fdfd0a8aab7574f648085d647817e7a90214cc0a30b7c14c47d3657"}
Nov 03 20:15:07.428303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.428283    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk" event={"ID":"d5f116fd-1e4c-4b58-b070-6c48e132562c","Type":"ContainerStarted","Data":"b9a5ee9fbc405c640a0f29acf4379f2795bbcb70e6d5237611773d7f8359b964"}
Nov 03 20:15:07.466510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.442083    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"]
Nov 03 20:15:07.528426 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.528389    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:15:07.529886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.529816    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 03 20:15:07.529886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.529870    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" podUID="cc874822-913a-4d2d-922f-19d6095c34f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 03 20:15:07.553075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.551771    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Nov 03 20:15:07.553075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.551838    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" podUID="15cce432-d79b-4f87-b5c6-63908192552d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Nov 03 20:15:07.574728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.574127    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:15:07.575018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.574983    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" podUID="992549e4-0f0c-4a0e-a170-a472285aa24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 03 20:15:07.582614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.582529    5659 generic.go:356] "Generic (PLEG): container finished" podID="180b9b66-9700-4b5e-bb2b-fb979d97a34b" containerID="8c9d4e8d8fdfd0a8aab7574f648085d647817e7a90214cc0a30b7c14c47d3657" exitCode=60
Nov 03 20:15:07.582881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.582850    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84" event={"ID":"180b9b66-9700-4b5e-bb2b-fb979d97a34b","Type":"ContainerDied","Data":"8c9d4e8d8fdfd0a8aab7574f648085d647817e7a90214cc0a30b7c14c47d3657"}
Nov 03 20:15:07.583616 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.583597    5659 scope.go:122] "RemoveContainer" containerID="8c9d4e8d8fdfd0a8aab7574f648085d647817e7a90214cc0a30b7c14c47d3657"
Nov 03 20:15:07.596512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.596458    5659 generic.go:356] "Generic (PLEG): container finished" podID="d5f116fd-1e4c-4b58-b070-6c48e132562c" containerID="1fe404c7e827a5f516b7c58f258141f222408ed83d2212f5488ce89b3410f19d" exitCode=60
Nov 03 20:15:07.597523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.596982    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk" event={"ID":"d5f116fd-1e4c-4b58-b070-6c48e132562c","Type":"ContainerDied","Data":"1fe404c7e827a5f516b7c58f258141f222408ed83d2212f5488ce89b3410f19d"}
Nov 03 20:15:07.598373 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.597789    5659 scope.go:122] "RemoveContainer" containerID="1fe404c7e827a5f516b7c58f258141f222408ed83d2212f5488ce89b3410f19d"
Nov 03 20:15:07.986237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:07.986193    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:08.530556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:08.530521    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:15:08.530947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:08.530924    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:15:08.531856 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:08.531827    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:15:08.540082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:08.540057    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:15:08.541075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:08.541055    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:15:08.541846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:08.541827    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:15:08.579910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:08.579880    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:15:08.580194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:08.580168    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:15:08.580944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:08.580924    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:15:08.610248 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:08.610201    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84" event={"ID":"180b9b66-9700-4b5e-bb2b-fb979d97a34b","Type":"ContainerStarted","Data":"50691312fb7441ae480e59977abdd0122717800ed509e8f5307b075423e236a3"}
Nov 03 20:15:08.623095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:08.622921    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk" event={"ID":"d5f116fd-1e4c-4b58-b070-6c48e132562c","Type":"ContainerStarted","Data":"0a51a6d29fe134b3ad829c5612b8cab39898b7ddff1551a4896bcfe67fee173d"}
Nov 03 20:15:08.666928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:08.665926    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk" podStartSLOduration=8.665907538 podStartE2EDuration="8.665907538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:08.66421651 +0000 UTC m=+2260.819118930" watchObservedRunningTime="2026-11-03 20:15:08.665907538 +0000 UTC m=+2260.820809958"
Nov 03 20:15:08.736564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:08.734045    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:09.632006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:09.631967    5659 generic.go:356] "Generic (PLEG): container finished" podID="180b9b66-9700-4b5e-bb2b-fb979d97a34b" containerID="50691312fb7441ae480e59977abdd0122717800ed509e8f5307b075423e236a3" exitCode=60
Nov 03 20:15:09.632553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:09.632069    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84" event={"ID":"180b9b66-9700-4b5e-bb2b-fb979d97a34b","Type":"ContainerDied","Data":"50691312fb7441ae480e59977abdd0122717800ed509e8f5307b075423e236a3"}
Nov 03 20:15:09.632553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:09.632129    5659 scope.go:122] "RemoveContainer" containerID="8c9d4e8d8fdfd0a8aab7574f648085d647817e7a90214cc0a30b7c14c47d3657"
Nov 03 20:15:09.632553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:09.632430    5659 scope.go:122] "RemoveContainer" containerID="50691312fb7441ae480e59977abdd0122717800ed509e8f5307b075423e236a3"
Nov 03 20:15:09.632904 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:09.632870    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84_odh-ai-gateway-infra(180b9b66-9700-4b5e-bb2b-fb979d97a34b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84" podUID="180b9b66-9700-4b5e-bb2b-fb979d97a34b"
Nov 03 20:15:09.635401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:09.635366    5659 generic.go:356] "Generic (PLEG): container finished" podID="d5f116fd-1e4c-4b58-b070-6c48e132562c" containerID="0a51a6d29fe134b3ad829c5612b8cab39898b7ddff1551a4896bcfe67fee173d" exitCode=60
Nov 03 20:15:09.635557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:09.635420    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk" event={"ID":"d5f116fd-1e4c-4b58-b070-6c48e132562c","Type":"ContainerDied","Data":"0a51a6d29fe134b3ad829c5612b8cab39898b7ddff1551a4896bcfe67fee173d"}
Nov 03 20:15:09.636028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:09.635920    5659 scope.go:122] "RemoveContainer" containerID="0a51a6d29fe134b3ad829c5612b8cab39898b7ddff1551a4896bcfe67fee173d"
Nov 03 20:15:09.637885 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:09.636317    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk_odh-ai-gateway-infra(d5f116fd-1e4c-4b58-b070-6c48e132562c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk" podUID="d5f116fd-1e4c-4b58-b070-6c48e132562c"
Nov 03 20:15:09.648513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:09.648493    5659 scope.go:122] "RemoveContainer" containerID="1fe404c7e827a5f516b7c58f258141f222408ed83d2212f5488ce89b3410f19d"
Nov 03 20:15:09.680047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:09.680012    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:10.221499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:10.221450    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"]
Nov 03 20:15:10.530654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:10.530574    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:10.649782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:10.649742    5659 scope.go:122] "RemoveContainer" containerID="50691312fb7441ae480e59977abdd0122717800ed509e8f5307b075423e236a3"
Nov 03 20:15:10.650275 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:10.650128    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84_odh-ai-gateway-infra(180b9b66-9700-4b5e-bb2b-fb979d97a34b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84" podUID="180b9b66-9700-4b5e-bb2b-fb979d97a34b"
Nov 03 20:15:10.651456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:10.651437    5659 scope.go:122] "RemoveContainer" containerID="0a51a6d29fe134b3ad829c5612b8cab39898b7ddff1551a4896bcfe67fee173d"
Nov 03 20:15:10.651783 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:10.651759    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk_odh-ai-gateway-infra(d5f116fd-1e4c-4b58-b070-6c48e132562c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk" podUID="d5f116fd-1e4c-4b58-b070-6c48e132562c"
Nov 03 20:15:11.262891 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:11.262843    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:11.927690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:11.927653    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:12.483047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.482998    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982" podUID="12961900-21e3-4340-8942-964f5588ae12" containerName="payload-processing" containerID="cri-o://dc7879ad6793f5d42c4efdc1b999e817954ffb844f12c8c21026b8a46c820c18" gracePeriod=25
Nov 03 20:15:12.728103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.728061    5659 generic.go:356] "Generic (PLEG): container finished" podID="12961900-21e3-4340-8942-964f5588ae12" containerID="dc7879ad6793f5d42c4efdc1b999e817954ffb844f12c8c21026b8a46c820c18" exitCode=0
Nov 03 20:15:12.728996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.728454    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982" event={"ID":"12961900-21e3-4340-8942-964f5588ae12","Type":"ContainerDied","Data":"dc7879ad6793f5d42c4efdc1b999e817954ffb844f12c8c21026b8a46c820c18"}
Nov 03 20:15:12.767591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.767384    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:12.932539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.932462    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:15:12.972707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.972669    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"]
Nov 03 20:15:12.974381 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.974353    5659 cpu_manager.go:414] "containerMap: removing container" podUID="12961900-21e3-4340-8942-964f5588ae12" containerName="payload-processing"
Nov 03 20:15:12.974550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.974536    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12961900-21e3-4340-8942-964f5588ae12" containerName="payload-processing"
Nov 03 20:15:12.974927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.974912    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="12961900-21e3-4340-8942-964f5588ae12" containerName="payload-processing"
Nov 03 20:15:12.987381 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.987346    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-plugins-config-volume\") pod \"12961900-21e3-4340-8942-964f5588ae12\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") "
Nov 03 20:15:12.987635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.987425    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-provider-credentials\" (UniqueName: \"kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-provider-credentials\") pod \"12961900-21e3-4340-8942-964f5588ae12\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") "
Nov 03 20:15:12.987635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.987599    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-routing-overlay-volume\") pod \"12961900-21e3-4340-8942-964f5588ae12\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") "
Nov 03 20:15:12.987778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.987721    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-kube-api-access-zrqx9\" (UniqueName: \"kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-kube-api-access-zrqx9\") pod \"12961900-21e3-4340-8942-964f5588ae12\" (UID: \"12961900-21e3-4340-8942-964f5588ae12\") "
Nov 03 20:15:12.989225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.988570    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-plugins-config-volume" pod "12961900-21e3-4340-8942-964f5588ae12" (UID: "12961900-21e3-4340-8942-964f5588ae12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:12.989225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.988852    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-routing-overlay-volume" pod "12961900-21e3-4340-8942-964f5588ae12" (UID: "12961900-21e3-4340-8942-964f5588ae12"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:12.989225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.989179    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:12.989225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.989202    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12961900-21e3-4340-8942-964f5588ae12-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:12.991928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.991908    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"]
Nov 03 20:15:12.994448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:12.992709    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"
Nov 03 20:15:13.000891 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.000867    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"
Nov 03 20:15:13.003033 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.002818    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"]
Nov 03 20:15:13.014281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.013167    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-provider-credentials" pod "12961900-21e3-4340-8942-964f5588ae12" (UID: "12961900-21e3-4340-8942-964f5588ae12"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:13.014281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.013224    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"]
Nov 03 20:15:13.019171 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.018650    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-kube-api-access-zrqx9" pod "12961900-21e3-4340-8942-964f5588ae12" (UID: "12961900-21e3-4340-8942-964f5588ae12"). InnerVolumeSpecName "kube-api-access-zrqx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:13.119150 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.119109    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvrx7\" (UniqueName: \"kubernetes.io/projected/3aa2c645-beca-46e7-a3af-fd1654bdb351-kube-api-access-hvrx7\") pod \"maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g\" (UID: \"3aa2c645-beca-46e7-a3af-fd1654bdb351\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"
Nov 03 20:15:13.119321 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.119171    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa2c645-beca-46e7-a3af-fd1654bdb351-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g\" (UID: \"3aa2c645-beca-46e7-a3af-fd1654bdb351\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"
Nov 03 20:15:13.119422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.119379    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkjb\" (UniqueName: \"kubernetes.io/projected/1601b767-8367-4476-a1b9-c33c58403370-kube-api-access-zhkjb\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh\" (UID: \"1601b767-8367-4476-a1b9-c33c58403370\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"
Nov 03 20:15:13.119647 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.119627    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1601b767-8367-4476-a1b9-c33c58403370-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh\" (UID: \"1601b767-8367-4476-a1b9-c33c58403370\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"
Nov 03 20:15:13.119929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.119814    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:13.119929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.119846    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrqx9\" (UniqueName: \"kubernetes.io/projected/12961900-21e3-4340-8942-964f5588ae12-kube-api-access-zrqx9\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:13.221541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.221502    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvrx7\" (UniqueName: \"kubernetes.io/projected/3aa2c645-beca-46e7-a3af-fd1654bdb351-kube-api-access-hvrx7\") pod \"maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g\" (UID: \"3aa2c645-beca-46e7-a3af-fd1654bdb351\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"
Nov 03 20:15:13.221541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.221549    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa2c645-beca-46e7-a3af-fd1654bdb351-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g\" (UID: \"3aa2c645-beca-46e7-a3af-fd1654bdb351\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"
Nov 03 20:15:13.221813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.221620    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkjb\" (UniqueName: \"kubernetes.io/projected/1601b767-8367-4476-a1b9-c33c58403370-kube-api-access-zhkjb\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh\" (UID: \"1601b767-8367-4476-a1b9-c33c58403370\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"
Nov 03 20:15:13.221813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.221799    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1601b767-8367-4476-a1b9-c33c58403370-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh\" (UID: \"1601b767-8367-4476-a1b9-c33c58403370\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"
Nov 03 20:15:13.222392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.222373    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa2c645-beca-46e7-a3af-fd1654bdb351-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g\" (UID: \"3aa2c645-beca-46e7-a3af-fd1654bdb351\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"
Nov 03 20:15:13.222454 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.222438    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1601b767-8367-4476-a1b9-c33c58403370-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh\" (UID: \"1601b767-8367-4476-a1b9-c33c58403370\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"
Nov 03 20:15:13.235187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.235157    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkjb\" (UniqueName: \"kubernetes.io/projected/1601b767-8367-4476-a1b9-c33c58403370-kube-api-access-zhkjb\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh\" (UID: \"1601b767-8367-4476-a1b9-c33c58403370\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"
Nov 03 20:15:13.235325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.235246    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvrx7\" (UniqueName: \"kubernetes.io/projected/3aa2c645-beca-46e7-a3af-fd1654bdb351-kube-api-access-hvrx7\") pod \"maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g\" (UID: \"3aa2c645-beca-46e7-a3af-fd1654bdb351\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"
Nov 03 20:15:13.397603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.397511    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"
Nov 03 20:15:13.442434 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.442399    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"
Nov 03 20:15:13.579634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.579399    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:13.700614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.700520    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"]
Nov 03 20:15:13.748984 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.748946    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982" event={"ID":"12961900-21e3-4340-8942-964f5588ae12","Type":"ContainerDied","Data":"f207a89b0fa633e14224792bb1dd227709d4c28fb056b76680a2508a776ab3f9"}
Nov 03 20:15:13.749142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.749021    5659 scope.go:122] "RemoveContainer" containerID="dc7879ad6793f5d42c4efdc1b999e817954ffb844f12c8c21026b8a46c820c18"
Nov 03 20:15:13.749235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.749219    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"
Nov 03 20:15:13.776626 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:15:13.776595    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1601b767_8367_4476_a1b9_c33c58403370.slice/crio-3aebf70dc58909505a1e407621c3b57898eabd6a81cabb33f2a7ccfca326eb29 WatchSource:0}: Error finding container 3aebf70dc58909505a1e407621c3b57898eabd6a81cabb33f2a7ccfca326eb29: Status 404 returned error can't find the container with id 3aebf70dc58909505a1e407621c3b57898eabd6a81cabb33f2a7ccfca326eb29
Nov 03 20:15:13.777042 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.777019    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"]
Nov 03 20:15:13.786665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.786635    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"]
Nov 03 20:15:13.791024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.791000    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-57d48g982"]
Nov 03 20:15:13.972744 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:13.972708    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12961900-21e3-4340-8942-964f5588ae12" path="/var/lib/kubelet/pods/12961900-21e3-4340-8942-964f5588ae12/volumes"
Nov 03 20:15:14.301178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:14.301144    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:14.761197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:14.761155    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g" event={"ID":"3aa2c645-beca-46e7-a3af-fd1654bdb351","Type":"ContainerStarted","Data":"c395ba1d87ec69e34cd58e9105b5e5d70d578ce048be114475c83cbf2f613893"}
Nov 03 20:15:14.761387 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:14.761222    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g" event={"ID":"3aa2c645-beca-46e7-a3af-fd1654bdb351","Type":"ContainerStarted","Data":"6d177ddd6e0195b83b2cae5d657d4de3e3bbbf58ec4efd3b3909b0c3e299c403"}
Nov 03 20:15:14.762878 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:14.762850    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh" event={"ID":"1601b767-8367-4476-a1b9-c33c58403370","Type":"ContainerStarted","Data":"913280b5bb0414c05a826358d2be3889d6cf5450c0c69fa14d9001353ed89769"}
Nov 03 20:15:14.762981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:14.762899    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh" event={"ID":"1601b767-8367-4476-a1b9-c33c58403370","Type":"ContainerStarted","Data":"3aebf70dc58909505a1e407621c3b57898eabd6a81cabb33f2a7ccfca326eb29"}
Nov 03 20:15:14.782401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:14.782351    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g" podStartSLOduration=14.78233446 podStartE2EDuration="14.78233446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:14.778268332 +0000 UTC m=+2266.933170752" watchObservedRunningTime="2026-11-03 20:15:14.78233446 +0000 UTC m=+2266.937236878"
Nov 03 20:15:14.797415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:14.797359    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh" podStartSLOduration=14.79734394 podStartE2EDuration="14.79734394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:14.796633275 +0000 UTC m=+2266.951535695" watchObservedRunningTime="2026-11-03 20:15:14.79734394 +0000 UTC m=+2266.952246360"
Nov 03 20:15:15.004908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.004862    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:15.219031 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:15.218995    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:15:15.222250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.222214    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv" podUID="229b2907-4221-408f-a677-bfbd3efcae60" containerName="payload-processing" containerID="cri-o://79a0e6e8ce5c389cf692345aee99877d3ba92c42e25b329a075a5c514e563f9d" gracePeriod=25
Nov 03 20:15:15.531473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.531445    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:15:15.681038 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.680946    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-provider-credentials\" (UniqueName: \"kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-provider-credentials\") pod \"229b2907-4221-408f-a677-bfbd3efcae60\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") "
Nov 03 20:15:15.681038 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.681003    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-routing-overlay-volume\") pod \"229b2907-4221-408f-a677-bfbd3efcae60\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") "
Nov 03 20:15:15.681296 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.681062    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-plugins-config-volume\") pod \"229b2907-4221-408f-a677-bfbd3efcae60\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") "
Nov 03 20:15:15.681296 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.681153    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-kube-api-access-7svls\" (UniqueName: \"kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-kube-api-access-7svls\") pod \"229b2907-4221-408f-a677-bfbd3efcae60\" (UID: \"229b2907-4221-408f-a677-bfbd3efcae60\") "
Nov 03 20:15:15.681495 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.681455    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-routing-overlay-volume" pod "229b2907-4221-408f-a677-bfbd3efcae60" (UID: "229b2907-4221-408f-a677-bfbd3efcae60"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:15.681654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.681493    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-plugins-config-volume" pod "229b2907-4221-408f-a677-bfbd3efcae60" (UID: "229b2907-4221-408f-a677-bfbd3efcae60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:15.682254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.682230    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:15.682330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.682257    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/229b2907-4221-408f-a677-bfbd3efcae60-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:15.683301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.683281    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-provider-credentials" pod "229b2907-4221-408f-a677-bfbd3efcae60" (UID: "229b2907-4221-408f-a677-bfbd3efcae60"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:15.683701 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.683678    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-kube-api-access-7svls" pod "229b2907-4221-408f-a677-bfbd3efcae60" (UID: "229b2907-4221-408f-a677-bfbd3efcae60"). InnerVolumeSpecName "kube-api-access-7svls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:15.763012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.762975    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:15.796522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.788256    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:15.796522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.788297    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7svls\" (UniqueName: \"kubernetes.io/projected/229b2907-4221-408f-a677-bfbd3efcae60-kube-api-access-7svls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:15.816568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.816528    5659 generic.go:356] "Generic (PLEG): container finished" podID="3aa2c645-beca-46e7-a3af-fd1654bdb351" containerID="c395ba1d87ec69e34cd58e9105b5e5d70d578ce048be114475c83cbf2f613893" exitCode=60
Nov 03 20:15:15.816797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.816690    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g" event={"ID":"3aa2c645-beca-46e7-a3af-fd1654bdb351","Type":"ContainerDied","Data":"c395ba1d87ec69e34cd58e9105b5e5d70d578ce048be114475c83cbf2f613893"}
Nov 03 20:15:15.817926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.817898    5659 scope.go:122] "RemoveContainer" containerID="c395ba1d87ec69e34cd58e9105b5e5d70d578ce048be114475c83cbf2f613893"
Nov 03 20:15:15.830888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.825023    5659 generic.go:356] "Generic (PLEG): container finished" podID="1601b767-8367-4476-a1b9-c33c58403370" containerID="913280b5bb0414c05a826358d2be3889d6cf5450c0c69fa14d9001353ed89769" exitCode=60
Nov 03 20:15:15.830888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.825260    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh" event={"ID":"1601b767-8367-4476-a1b9-c33c58403370","Type":"ContainerDied","Data":"913280b5bb0414c05a826358d2be3889d6cf5450c0c69fa14d9001353ed89769"}
Nov 03 20:15:15.830888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.825989    5659 scope.go:122] "RemoveContainer" containerID="913280b5bb0414c05a826358d2be3889d6cf5450c0c69fa14d9001353ed89769"
Nov 03 20:15:15.836247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.836212    5659 generic.go:356] "Generic (PLEG): container finished" podID="229b2907-4221-408f-a677-bfbd3efcae60" containerID="79a0e6e8ce5c389cf692345aee99877d3ba92c42e25b329a075a5c514e563f9d" exitCode=0
Nov 03 20:15:15.836427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.836317    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv" event={"ID":"229b2907-4221-408f-a677-bfbd3efcae60","Type":"ContainerDied","Data":"79a0e6e8ce5c389cf692345aee99877d3ba92c42e25b329a075a5c514e563f9d"}
Nov 03 20:15:15.836427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.836372    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv" event={"ID":"229b2907-4221-408f-a677-bfbd3efcae60","Type":"ContainerDied","Data":"b64cc69364c62a857c77466ab812885f568cba140c368ad6a7adb44af4e4078b"}
Nov 03 20:15:15.836598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.836463    5659 scope.go:122] "RemoveContainer" containerID="79a0e6e8ce5c389cf692345aee99877d3ba92c42e25b329a075a5c514e563f9d"
Nov 03 20:15:15.836757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.836726    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"
Nov 03 20:15:15.896624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.896600    5659 scope.go:122] "RemoveContainer" containerID="79a0e6e8ce5c389cf692345aee99877d3ba92c42e25b329a075a5c514e563f9d"
Nov 03 20:15:15.897262 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:15.897234    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a0e6e8ce5c389cf692345aee99877d3ba92c42e25b329a075a5c514e563f9d\": container with ID starting with 79a0e6e8ce5c389cf692345aee99877d3ba92c42e25b329a075a5c514e563f9d not found: ID does not exist" containerID="79a0e6e8ce5c389cf692345aee99877d3ba92c42e25b329a075a5c514e563f9d"
Nov 03 20:15:15.897424 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.897400    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a0e6e8ce5c389cf692345aee99877d3ba92c42e25b329a075a5c514e563f9d"} err="failed to get container status \"79a0e6e8ce5c389cf692345aee99877d3ba92c42e25b329a075a5c514e563f9d\": rpc error: code = NotFound desc = could not find container \"79a0e6e8ce5c389cf692345aee99877d3ba92c42e25b329a075a5c514e563f9d\": container with ID starting with 79a0e6e8ce5c389cf692345aee99877d3ba92c42e25b329a075a5c514e563f9d not found: ID does not exist"
Nov 03 20:15:15.917182 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.917154    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"]
Nov 03 20:15:15.923432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.923406    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bxqhkv"]
Nov 03 20:15:15.977379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:15.977347    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="229b2907-4221-408f-a677-bfbd3efcae60" path="/var/lib/kubelet/pods/229b2907-4221-408f-a677-bfbd3efcae60/volumes"
Nov 03 20:15:16.734810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:16.734776    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:16.852216 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:16.852179    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh" event={"ID":"1601b767-8367-4476-a1b9-c33c58403370","Type":"ContainerStarted","Data":"58b96ad0d3fb0c7bf9c23269f04bad8fc305acadd12d52d8845fde3a2196f268"}
Nov 03 20:15:16.852792 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:16.852772    5659 scope.go:122] "RemoveContainer" containerID="58b96ad0d3fb0c7bf9c23269f04bad8fc305acadd12d52d8845fde3a2196f268"
Nov 03 20:15:16.853270 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:16.853247    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh_odh-ai-gateway-infra(1601b767-8367-4476-a1b9-c33c58403370)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh" podUID="1601b767-8367-4476-a1b9-c33c58403370"
Nov 03 20:15:16.857905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:16.857881    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g" event={"ID":"3aa2c645-beca-46e7-a3af-fd1654bdb351","Type":"ContainerStarted","Data":"fe6821510064fc95458a4d835538ee3b2bd03c22e025bb3007462d179033c80e"}
Nov 03 20:15:17.627509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:17.627445    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:17.871669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:17.871633    5659 generic.go:356] "Generic (PLEG): container finished" podID="3aa2c645-beca-46e7-a3af-fd1654bdb351" containerID="fe6821510064fc95458a4d835538ee3b2bd03c22e025bb3007462d179033c80e" exitCode=60
Nov 03 20:15:17.872187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:17.871715    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g" event={"ID":"3aa2c645-beca-46e7-a3af-fd1654bdb351","Type":"ContainerDied","Data":"fe6821510064fc95458a4d835538ee3b2bd03c22e025bb3007462d179033c80e"}
Nov 03 20:15:17.872187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:17.871785    5659 scope.go:122] "RemoveContainer" containerID="c395ba1d87ec69e34cd58e9105b5e5d70d578ce048be114475c83cbf2f613893"
Nov 03 20:15:17.872187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:17.872089    5659 scope.go:122] "RemoveContainer" containerID="fe6821510064fc95458a4d835538ee3b2bd03c22e025bb3007462d179033c80e"
Nov 03 20:15:17.872601 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:17.872563    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g_odh-ai-gateway-infra(3aa2c645-beca-46e7-a3af-fd1654bdb351)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g" podUID="3aa2c645-beca-46e7-a3af-fd1654bdb351"
Nov 03 20:15:17.874964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:17.874939    5659 generic.go:356] "Generic (PLEG): container finished" podID="1601b767-8367-4476-a1b9-c33c58403370" containerID="58b96ad0d3fb0c7bf9c23269f04bad8fc305acadd12d52d8845fde3a2196f268" exitCode=60
Nov 03 20:15:17.875073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:17.874968    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh" event={"ID":"1601b767-8367-4476-a1b9-c33c58403370","Type":"ContainerDied","Data":"58b96ad0d3fb0c7bf9c23269f04bad8fc305acadd12d52d8845fde3a2196f268"}
Nov 03 20:15:17.875404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:17.875387    5659 scope.go:122] "RemoveContainer" containerID="58b96ad0d3fb0c7bf9c23269f04bad8fc305acadd12d52d8845fde3a2196f268"
Nov 03 20:15:17.875907 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:17.875878    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh_odh-ai-gateway-infra(1601b767-8367-4476-a1b9-c33c58403370)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh" podUID="1601b767-8367-4476-a1b9-c33c58403370"
Nov 03 20:15:17.886840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:17.886823    5659 scope.go:122] "RemoveContainer" containerID="913280b5bb0414c05a826358d2be3889d6cf5450c0c69fa14d9001353ed89769"
Nov 03 20:15:18.445532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:18.442247    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:18.888017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:18.887982    5659 scope.go:122] "RemoveContainer" containerID="fe6821510064fc95458a4d835538ee3b2bd03c22e025bb3007462d179033c80e"
Nov 03 20:15:18.888527 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:18.888406    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g_odh-ai-gateway-infra(3aa2c645-beca-46e7-a3af-fd1654bdb351)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g" podUID="3aa2c645-beca-46e7-a3af-fd1654bdb351"
Nov 03 20:15:18.889802 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:18.889780    5659 scope.go:122] "RemoveContainer" containerID="58b96ad0d3fb0c7bf9c23269f04bad8fc305acadd12d52d8845fde3a2196f268"
Nov 03 20:15:18.890209 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:18.890151    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh_odh-ai-gateway-infra(1601b767-8367-4476-a1b9-c33c58403370)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh" podUID="1601b767-8367-4476-a1b9-c33c58403370"
Nov 03 20:15:19.251726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:19.251691    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:19.963071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:19.963029    5659 scope.go:122] "RemoveContainer" containerID="0a51a6d29fe134b3ad829c5612b8cab39898b7ddff1551a4896bcfe67fee173d"
Nov 03 20:15:20.059716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:20.059681    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:20.785997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:20.785927    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:20.912663 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:20.912624    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk" event={"ID":"d5f116fd-1e4c-4b58-b070-6c48e132562c","Type":"ContainerStarted","Data":"c80fb7ac40a78ab0a7f670441665bbae21a58746d7d1b0b9192f32ce988d2a61"}
Nov 03 20:15:21.564678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:21.564642    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:21.923018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:21.922923    5659 generic.go:356] "Generic (PLEG): container finished" podID="d5f116fd-1e4c-4b58-b070-6c48e132562c" containerID="c80fb7ac40a78ab0a7f670441665bbae21a58746d7d1b0b9192f32ce988d2a61" exitCode=60
Nov 03 20:15:21.923018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:21.922995    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk" event={"ID":"d5f116fd-1e4c-4b58-b070-6c48e132562c","Type":"ContainerDied","Data":"c80fb7ac40a78ab0a7f670441665bbae21a58746d7d1b0b9192f32ce988d2a61"}
Nov 03 20:15:21.923268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:21.923056    5659 scope.go:122] "RemoveContainer" containerID="0a51a6d29fe134b3ad829c5612b8cab39898b7ddff1551a4896bcfe67fee173d"
Nov 03 20:15:21.923507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:21.923473    5659 scope.go:122] "RemoveContainer" containerID="c80fb7ac40a78ab0a7f670441665bbae21a58746d7d1b0b9192f32ce988d2a61"
Nov 03 20:15:21.923840 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:21.923815    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk_odh-ai-gateway-infra(d5f116fd-1e4c-4b58-b070-6c48e132562c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk" podUID="d5f116fd-1e4c-4b58-b070-6c48e132562c"
Nov 03 20:15:21.961945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:21.961902    5659 scope.go:122] "RemoveContainer" containerID="50691312fb7441ae480e59977abdd0122717800ed509e8f5307b075423e236a3"
Nov 03 20:15:22.007876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:22.007847    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"]
Nov 03 20:15:22.463272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:22.463219    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:22.935750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:22.935719    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84" event={"ID":"180b9b66-9700-4b5e-bb2b-fb979d97a34b","Type":"ContainerStarted","Data":"fe3ef447f38eb492609a9e98e75ba9d4da5f818db5295c58a68386557c3b0a76"}
Nov 03 20:15:23.128069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.128042    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"
Nov 03 20:15:23.169697 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.169663    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:23.249128 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.249082    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5f116fd-1e4c-4b58-b070-6c48e132562c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f116fd-1e4c-4b58-b070-6c48e132562c-maas-api-service-ca\") pod \"d5f116fd-1e4c-4b58-b070-6c48e132562c\" (UID: \"d5f116fd-1e4c-4b58-b070-6c48e132562c\") "
Nov 03 20:15:23.249459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.249435    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f116fd-1e4c-4b58-b070-6c48e132562c-kube-api-access-zx5ct\" (UniqueName: \"kubernetes.io/projected/d5f116fd-1e4c-4b58-b070-6c48e132562c-kube-api-access-zx5ct\") pod \"d5f116fd-1e4c-4b58-b070-6c48e132562c\" (UID: \"d5f116fd-1e4c-4b58-b070-6c48e132562c\") "
Nov 03 20:15:23.250148 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.250112    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f116fd-1e4c-4b58-b070-6c48e132562c-maas-api-service-ca" pod "d5f116fd-1e4c-4b58-b070-6c48e132562c" (UID: "d5f116fd-1e4c-4b58-b070-6c48e132562c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:23.250512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.250469    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f116fd-1e4c-4b58-b070-6c48e132562c-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:23.289825 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.289781    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f116fd-1e4c-4b58-b070-6c48e132562c-kube-api-access-zx5ct" pod "d5f116fd-1e4c-4b58-b070-6c48e132562c" (UID: "d5f116fd-1e4c-4b58-b070-6c48e132562c"). InnerVolumeSpecName "kube-api-access-zx5ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:23.352108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.352075    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx5ct\" (UniqueName: \"kubernetes.io/projected/d5f116fd-1e4c-4b58-b070-6c48e132562c-kube-api-access-zx5ct\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:23.867465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.867423    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:23.976168 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.976130    5659 generic.go:356] "Generic (PLEG): container finished" podID="180b9b66-9700-4b5e-bb2b-fb979d97a34b" containerID="fe3ef447f38eb492609a9e98e75ba9d4da5f818db5295c58a68386557c3b0a76" exitCode=60
Nov 03 20:15:23.978157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.978136    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"
Nov 03 20:15:23.980248 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.980214    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"]
Nov 03 20:15:23.980637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.980274    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84" event={"ID":"180b9b66-9700-4b5e-bb2b-fb979d97a34b","Type":"ContainerDied","Data":"fe3ef447f38eb492609a9e98e75ba9d4da5f818db5295c58a68386557c3b0a76"}
Nov 03 20:15:23.980637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.980310    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk" event={"ID":"d5f116fd-1e4c-4b58-b070-6c48e132562c","Type":"ContainerDied","Data":"b9a5ee9fbc405c640a0f29acf4379f2795bbcb70e6d5237611773d7f8359b964"}
Nov 03 20:15:23.980637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:23.980345    5659 scope.go:122] "RemoveContainer" containerID="50691312fb7441ae480e59977abdd0122717800ed509e8f5307b075423e236a3"
Nov 03 20:15:24.002334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.002313    5659 scope.go:122] "RemoveContainer" containerID="c80fb7ac40a78ab0a7f670441665bbae21a58746d7d1b0b9192f32ce988d2a61"
Nov 03 20:15:24.025153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.025121    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"]
Nov 03 20:15:24.030296 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.030268    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895615-gh8dk"]
Nov 03 20:15:24.198451 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.198424    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"
Nov 03 20:15:24.264718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.264686    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"]
Nov 03 20:15:24.265964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.265940    5659 cpu_manager.go:414] "containerMap: removing container" podUID="180b9b66-9700-4b5e-bb2b-fb979d97a34b" containerName="cleanup"
Nov 03 20:15:24.266031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.265966    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180b9b66-9700-4b5e-bb2b-fb979d97a34b" containerName="cleanup"
Nov 03 20:15:24.266031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266000    5659 cpu_manager.go:414] "containerMap: removing container" podUID="180b9b66-9700-4b5e-bb2b-fb979d97a34b" containerName="cleanup"
Nov 03 20:15:24.266031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266009    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180b9b66-9700-4b5e-bb2b-fb979d97a34b" containerName="cleanup"
Nov 03 20:15:24.266031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266023    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d5f116fd-1e4c-4b58-b070-6c48e132562c" containerName="cleanup"
Nov 03 20:15:24.266232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266030    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f116fd-1e4c-4b58-b070-6c48e132562c" containerName="cleanup"
Nov 03 20:15:24.266232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266045    5659 cpu_manager.go:414] "containerMap: removing container" podUID="229b2907-4221-408f-a677-bfbd3efcae60" containerName="payload-processing"
Nov 03 20:15:24.266232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266053    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="229b2907-4221-408f-a677-bfbd3efcae60" containerName="payload-processing"
Nov 03 20:15:24.266232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266078    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d5f116fd-1e4c-4b58-b070-6c48e132562c" containerName="cleanup"
Nov 03 20:15:24.266232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266085    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f116fd-1e4c-4b58-b070-6c48e132562c" containerName="cleanup"
Nov 03 20:15:24.266232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266122    5659 cpu_manager.go:414] "containerMap: removing container" podUID="180b9b66-9700-4b5e-bb2b-fb979d97a34b" containerName="cleanup"
Nov 03 20:15:24.266232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266129    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180b9b66-9700-4b5e-bb2b-fb979d97a34b" containerName="cleanup"
Nov 03 20:15:24.266450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266362    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="180b9b66-9700-4b5e-bb2b-fb979d97a34b" containerName="cleanup"
Nov 03 20:15:24.266450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266380    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f116fd-1e4c-4b58-b070-6c48e132562c" containerName="cleanup"
Nov 03 20:15:24.266450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266392    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="180b9b66-9700-4b5e-bb2b-fb979d97a34b" containerName="cleanup"
Nov 03 20:15:24.266450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266405    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="229b2907-4221-408f-a677-bfbd3efcae60" containerName="payload-processing"
Nov 03 20:15:24.266450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.266418    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f116fd-1e4c-4b58-b070-6c48e132562c" containerName="cleanup"
Nov 03 20:15:24.271104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.271078    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.272127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.272097    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"]
Nov 03 20:15:24.280092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.280065    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/180b9b66-9700-4b5e-bb2b-fb979d97a34b-kube-api-access-mvd2f\" (UniqueName: \"kubernetes.io/projected/180b9b66-9700-4b5e-bb2b-fb979d97a34b-kube-api-access-mvd2f\") pod \"180b9b66-9700-4b5e-bb2b-fb979d97a34b\" (UID: \"180b9b66-9700-4b5e-bb2b-fb979d97a34b\") "
Nov 03 20:15:24.280241 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.280114    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/180b9b66-9700-4b5e-bb2b-fb979d97a34b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180b9b66-9700-4b5e-bb2b-fb979d97a34b-maas-api-service-ca\") pod \"180b9b66-9700-4b5e-bb2b-fb979d97a34b\" (UID: \"180b9b66-9700-4b5e-bb2b-fb979d97a34b\") "
Nov 03 20:15:24.281458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.281375    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180b9b66-9700-4b5e-bb2b-fb979d97a34b-maas-api-service-ca" pod "180b9b66-9700-4b5e-bb2b-fb979d97a34b" (UID: "180b9b66-9700-4b5e-bb2b-fb979d97a34b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:24.285003 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.284978    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180b9b66-9700-4b5e-bb2b-fb979d97a34b-kube-api-access-mvd2f" pod "180b9b66-9700-4b5e-bb2b-fb979d97a34b" (UID: "180b9b66-9700-4b5e-bb2b-fb979d97a34b"). InnerVolumeSpecName "kube-api-access-mvd2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:24.382005 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.381965    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.382187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.382017    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.382278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.382231    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqslh\" (UniqueName: \"kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-kube-api-access-lqslh\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.382331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.382284    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.382387 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.382365    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.382425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.382402    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.382552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.382533    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.382639 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.382622    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.382757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.382744    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.382943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.382926    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvd2f\" (UniqueName: \"kubernetes.io/projected/180b9b66-9700-4b5e-bb2b-fb979d97a34b-kube-api-access-mvd2f\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:24.383000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.382948    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180b9b66-9700-4b5e-bb2b-fb979d97a34b-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:24.484109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.484073    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.484332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.484134    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.484332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.484266    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqslh\" (UniqueName: \"kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-kube-api-access-lqslh\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.484332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.484304    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.484529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.484341    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.484529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.484378    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.484529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.484468    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.484693 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.484604    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.484765 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.484710    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.484868 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.484803    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.484999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.484964    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.485133 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.485068    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.485133 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.485123    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.485443 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.485421    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.486649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.486627    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.486802 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.486780    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.496672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.496650    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.496788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.496738    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqslh\" (UniqueName: \"kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-kube-api-access-lqslh\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.590727 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.590689    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:24.763811 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.763687    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:24.892725 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:15:24.892696    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a091b5_503d_4f6a_8864_a02283bfb3c1.slice/crio-5c03e046e91be0d8b027c44cc752b3f1ad9ff868769161cd9f7f0d1e721adf1b WatchSource:0}: Error finding container 5c03e046e91be0d8b027c44cc752b3f1ad9ff868769161cd9f7f0d1e721adf1b: Status 404 returned error can't find the container with id 5c03e046e91be0d8b027c44cc752b3f1ad9ff868769161cd9f7f0d1e721adf1b
Nov 03 20:15:24.893561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.893536    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"]
Nov 03 20:15:24.895802 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.895769    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:15:24.895871 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.895836    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:15:24.895909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:24.895884    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:15:25.001845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.001815    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"
Nov 03 20:15:25.001845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.001822    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84" event={"ID":"180b9b66-9700-4b5e-bb2b-fb979d97a34b","Type":"ContainerDied","Data":"7989e1a08e6fe79887ad5605602ac91aedb3ba80c863c9f1fe3e78632893943a"}
Nov 03 20:15:25.002362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.001922    5659 scope.go:122] "RemoveContainer" containerID="fe3ef447f38eb492609a9e98e75ba9d4da5f818db5295c58a68386557c3b0a76"
Nov 03 20:15:25.005084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.005051    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" event={"ID":"c6a091b5-503d-4f6a-8864-a02283bfb3c1","Type":"ContainerStarted","Data":"0404f47c6bca8766762ed51e6ab6b0976667dcf1bca9de951952290a5bb97e32"}
Nov 03 20:15:25.005223 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.005113    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" event={"ID":"c6a091b5-503d-4f6a-8864-a02283bfb3c1","Type":"ContainerStarted","Data":"5c03e046e91be0d8b027c44cc752b3f1ad9ff868769161cd9f7f0d1e721adf1b"}
Nov 03 20:15:25.036022 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.035961    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" podStartSLOduration=12.035942679 podStartE2EDuration="12.035942679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:25.032135911 +0000 UTC m=+2277.187038348" watchObservedRunningTime="2026-11-03 20:15:25.035942679 +0000 UTC m=+2277.190845098"
Nov 03 20:15:25.052792 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.052748    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"]
Nov 03 20:15:25.060039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.060007    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895615-d7b84"]
Nov 03 20:15:25.591300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.591261    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:25.592851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.592823    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 20:15:25.593006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.592875    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" podUID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 20:15:25.630739 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.630703    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:25.974511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.974461    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="180b9b66-9700-4b5e-bb2b-fb979d97a34b" path="/var/lib/kubelet/pods/180b9b66-9700-4b5e-bb2b-fb979d97a34b/volumes"
Nov 03 20:15:25.975079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:25.975061    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f116fd-1e4c-4b58-b070-6c48e132562c" path="/var/lib/kubelet/pods/d5f116fd-1e4c-4b58-b070-6c48e132562c/volumes"
Nov 03 20:15:26.294131 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:26.293957    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:26.591804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:26.591702    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 20:15:26.591804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:26.591770    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" podUID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 20:15:27.052400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:27.052361    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:27.600506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:27.600035    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 20:15:27.600506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:27.600100    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" podUID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 20:15:28.214358 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:28.214129    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:28.513597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:28.504213    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:28.596509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:28.595899    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 20:15:28.596509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:28.595982    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" podUID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 20:15:28.639507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:28.637969    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:28.969557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:28.962520    5659 scope.go:122] "RemoveContainer" containerID="fe6821510064fc95458a4d835538ee3b2bd03c22e025bb3007462d179033c80e"
Nov 03 20:15:29.198543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:29.198507    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:29.591964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:29.591922    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 20:15:29.592256 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:29.592212    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" podUID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 20:15:29.911064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:29.910994    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:30.076960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:30.076526    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:30.203057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:30.203021    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g" event={"ID":"3aa2c645-beca-46e7-a3af-fd1654bdb351","Type":"ContainerStarted","Data":"8709c8d8a3220f042c2f51b7587f989d2d3e3151485dd7cbc592265ecfd74d50"}
Nov 03 20:15:30.593529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:30.591332    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 20:15:30.593529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:30.591398    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" podUID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 20:15:30.644053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:30.643449    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:30.918673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:30.918177    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:31.325521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.291956    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:31.332509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.332456    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"]
Nov 03 20:15:31.381206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.381174    5659 scope.go:122] "RemoveContainer" containerID="58b96ad0d3fb0c7bf9c23269f04bad8fc305acadd12d52d8845fde3a2196f268"
Nov 03 20:15:31.391574 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.391197    5659 generic.go:356] "Generic (PLEG): container finished" podID="3aa2c645-beca-46e7-a3af-fd1654bdb351" containerID="8709c8d8a3220f042c2f51b7587f989d2d3e3151485dd7cbc592265ecfd74d50" exitCode=60
Nov 03 20:15:31.391574 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.391563    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g" event={"ID":"3aa2c645-beca-46e7-a3af-fd1654bdb351","Type":"ContainerDied","Data":"8709c8d8a3220f042c2f51b7587f989d2d3e3151485dd7cbc592265ecfd74d50"}
Nov 03 20:15:31.391778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.391646    5659 scope.go:122] "RemoveContainer" containerID="fe6821510064fc95458a4d835538ee3b2bd03c22e025bb3007462d179033c80e"
Nov 03 20:15:31.594437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.593154    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 20:15:31.594437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.593222    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" podUID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 20:15:31.717736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.717702    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"
Nov 03 20:15:31.890378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.890283    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa2c645-beca-46e7-a3af-fd1654bdb351-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa2c645-beca-46e7-a3af-fd1654bdb351-maas-api-service-ca\") pod \"3aa2c645-beca-46e7-a3af-fd1654bdb351\" (UID: \"3aa2c645-beca-46e7-a3af-fd1654bdb351\") "
Nov 03 20:15:31.890643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.890619    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa2c645-beca-46e7-a3af-fd1654bdb351-maas-api-service-ca" pod "3aa2c645-beca-46e7-a3af-fd1654bdb351" (UID: "3aa2c645-beca-46e7-a3af-fd1654bdb351"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:31.890738 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.890628    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa2c645-beca-46e7-a3af-fd1654bdb351-kube-api-access-hvrx7\" (UniqueName: \"kubernetes.io/projected/3aa2c645-beca-46e7-a3af-fd1654bdb351-kube-api-access-hvrx7\") pod \"3aa2c645-beca-46e7-a3af-fd1654bdb351\" (UID: \"3aa2c645-beca-46e7-a3af-fd1654bdb351\") "
Nov 03 20:15:31.891184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.891166    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa2c645-beca-46e7-a3af-fd1654bdb351-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:31.893277 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.893247    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa2c645-beca-46e7-a3af-fd1654bdb351-kube-api-access-hvrx7" pod "3aa2c645-beca-46e7-a3af-fd1654bdb351" (UID: "3aa2c645-beca-46e7-a3af-fd1654bdb351"). InnerVolumeSpecName "kube-api-access-hvrx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:31.992690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:31.992659    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvrx7\" (UniqueName: \"kubernetes.io/projected/3aa2c645-beca-46e7-a3af-fd1654bdb351-kube-api-access-hvrx7\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:32.336504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:32.336072    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:32.401033 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:32.401006    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"
Nov 03 20:15:32.401237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:32.401057    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g" event={"ID":"3aa2c645-beca-46e7-a3af-fd1654bdb351","Type":"ContainerDied","Data":"6d177ddd6e0195b83b2cae5d657d4de3e3bbbf58ec4efd3b3909b0c3e299c403"}
Nov 03 20:15:32.401237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:32.401127    5659 scope.go:122] "RemoveContainer" containerID="8709c8d8a3220f042c2f51b7587f989d2d3e3151485dd7cbc592265ecfd74d50"
Nov 03 20:15:32.403786 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:32.403756    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh" event={"ID":"1601b767-8367-4476-a1b9-c33c58403370","Type":"ContainerStarted","Data":"63eb9d4a214b69ffa9a4138693fcfd96cd46f064a1806bbd0450d0b8df0a8d05"}
Nov 03 20:15:32.444458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:32.444420    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"]
Nov 03 20:15:32.457497 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:32.457459    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3fbb1a-29895615-nd44g"]
Nov 03 20:15:32.973718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:32.973674    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:32.976539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:32.974466    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:32.976539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:32.976098    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:15:33.157526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:33.157384    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:33.416135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:33.416037    5659 generic.go:356] "Generic (PLEG): container finished" podID="1601b767-8367-4476-a1b9-c33c58403370" containerID="63eb9d4a214b69ffa9a4138693fcfd96cd46f064a1806bbd0450d0b8df0a8d05" exitCode=60
Nov 03 20:15:33.416135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:33.416098    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh" event={"ID":"1601b767-8367-4476-a1b9-c33c58403370","Type":"ContainerDied","Data":"63eb9d4a214b69ffa9a4138693fcfd96cd46f064a1806bbd0450d0b8df0a8d05"}
Nov 03 20:15:33.416381 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:33.416157    5659 scope.go:122] "RemoveContainer" containerID="58b96ad0d3fb0c7bf9c23269f04bad8fc305acadd12d52d8845fde3a2196f268"
Nov 03 20:15:33.416646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:33.416630    5659 scope.go:122] "RemoveContainer" containerID="63eb9d4a214b69ffa9a4138693fcfd96cd46f064a1806bbd0450d0b8df0a8d05"
Nov 03 20:15:33.417077 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:33.417050    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh_odh-ai-gateway-infra(1601b767-8367-4476-a1b9-c33c58403370)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh" podUID="1601b767-8367-4476-a1b9-c33c58403370"
Nov 03 20:15:33.436691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:33.436664    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"]
Nov 03 20:15:33.874941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:33.874898    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:33.977026 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:33.976992    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa2c645-beca-46e7-a3af-fd1654bdb351" path="/var/lib/kubelet/pods/3aa2c645-beca-46e7-a3af-fd1654bdb351/volumes"
Nov 03 20:15:34.547784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:34.546545    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:34.684673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:34.684602    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"
Nov 03 20:15:34.809445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:34.809409    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1601b767-8367-4476-a1b9-c33c58403370-kube-api-access-zhkjb\" (UniqueName: \"kubernetes.io/projected/1601b767-8367-4476-a1b9-c33c58403370-kube-api-access-zhkjb\") pod \"1601b767-8367-4476-a1b9-c33c58403370\" (UID: \"1601b767-8367-4476-a1b9-c33c58403370\") "
Nov 03 20:15:34.809677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:34.809635    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1601b767-8367-4476-a1b9-c33c58403370-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1601b767-8367-4476-a1b9-c33c58403370-maas-api-service-ca\") pod \"1601b767-8367-4476-a1b9-c33c58403370\" (UID: \"1601b767-8367-4476-a1b9-c33c58403370\") "
Nov 03 20:15:34.810018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:34.809988    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1601b767-8367-4476-a1b9-c33c58403370-maas-api-service-ca" pod "1601b767-8367-4476-a1b9-c33c58403370" (UID: "1601b767-8367-4476-a1b9-c33c58403370"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:34.810673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:34.810650    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1601b767-8367-4476-a1b9-c33c58403370-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:34.811997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:34.811976    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1601b767-8367-4476-a1b9-c33c58403370-kube-api-access-zhkjb" pod "1601b767-8367-4476-a1b9-c33c58403370" (UID: "1601b767-8367-4476-a1b9-c33c58403370"). InnerVolumeSpecName "kube-api-access-zhkjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:34.912056 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:34.912021    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhkjb\" (UniqueName: \"kubernetes.io/projected/1601b767-8367-4476-a1b9-c33c58403370-kube-api-access-zhkjb\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:35.257523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:35.257446    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:35.440309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:35.440275    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"
Nov 03 20:15:35.440309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:35.440293    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh" event={"ID":"1601b767-8367-4476-a1b9-c33c58403370","Type":"ContainerDied","Data":"3aebf70dc58909505a1e407621c3b57898eabd6a81cabb33f2a7ccfca326eb29"}
Nov 03 20:15:35.440638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:35.440360    5659 scope.go:122] "RemoveContainer" containerID="63eb9d4a214b69ffa9a4138693fcfd96cd46f064a1806bbd0450d0b8df0a8d05"
Nov 03 20:15:35.470432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:35.470391    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"]
Nov 03 20:15:35.477764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:35.477728    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895615-5tjnh"]
Nov 03 20:15:35.952678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:35.952639    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:36.030754 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:36.030715    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1601b767-8367-4476-a1b9-c33c58403370" path="/var/lib/kubelet/pods/1601b767-8367-4476-a1b9-c33c58403370/volumes"
Nov 03 20:15:36.601210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:36.601166    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:36.737816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:36.734913    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:37.427451 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:37.427416    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:38.219959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:38.219923    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:38.661149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:38.661056    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:38.956457 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:38.956424    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:39.634049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:39.634013    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:40.335011 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:40.334969    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:41.139866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:41.139833    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:41.918184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:41.918145    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:42.050447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:42.050413    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:42.211846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:42.211807    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:43.032509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:43.032464    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:43.074941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:43.074907    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"]
Nov 03 20:15:43.184269 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:43.184234    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:43.425038 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:43.424919    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:43.818875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:43.818787    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:44.566078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:44.566044    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:44.719387 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:44.719352    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:45.224611 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:45.224571    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:15:45.252646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:45.252615    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:46.207745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:46.207686    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:46.941555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:46.941512    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:47.633281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:47.633212    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:48.075671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.075628    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr" podUID="dc97ad46-4774-4efa-a9fd-cb061eaea0d9" containerName="payload-processing" containerID="cri-o://0acd073c614eb322c4ee43b7063e24fe4fbba35d093769519ef64d2f27c13c7b" gracePeriod=25
Nov 03 20:15:48.331181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.331094    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:48.444557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.437126    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:15:48.469902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.469866    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"]
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.470947    5659 cpu_manager.go:414] "containerMap: removing container" podUID="dc97ad46-4774-4efa-a9fd-cb061eaea0d9" containerName="payload-processing"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.470975    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc97ad46-4774-4efa-a9fd-cb061eaea0d9" containerName="payload-processing"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471025    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3aa2c645-beca-46e7-a3af-fd1654bdb351" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471033    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa2c645-beca-46e7-a3af-fd1654bdb351" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471062    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d5f116fd-1e4c-4b58-b070-6c48e132562c" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471070    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f116fd-1e4c-4b58-b070-6c48e132562c" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471084    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3aa2c645-beca-46e7-a3af-fd1654bdb351" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471091    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa2c645-beca-46e7-a3af-fd1654bdb351" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471130    5659 cpu_manager.go:414] "containerMap: removing container" podUID="1601b767-8367-4476-a1b9-c33c58403370" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471138    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1601b767-8367-4476-a1b9-c33c58403370" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471151    5659 cpu_manager.go:414] "containerMap: removing container" podUID="1601b767-8367-4476-a1b9-c33c58403370" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471158    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1601b767-8367-4476-a1b9-c33c58403370" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471385    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f116fd-1e4c-4b58-b070-6c48e132562c" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471402    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa2c645-beca-46e7-a3af-fd1654bdb351" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471417    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="1601b767-8367-4476-a1b9-c33c58403370" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471428    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="1601b767-8367-4476-a1b9-c33c58403370" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471441    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc97ad46-4774-4efa-a9fd-cb061eaea0d9" containerName="payload-processing"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471453    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="180b9b66-9700-4b5e-bb2b-fb979d97a34b" containerName="cleanup"
Nov 03 20:15:48.474192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.471466    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa2c645-beca-46e7-a3af-fd1654bdb351" containerName="cleanup"
Nov 03 20:15:48.476047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.476024    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.484134 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.484111    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 20:15:48.484437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.484419    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-001b5c9f6a6a\""
Nov 03 20:15:48.484598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.484444    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 20:15:48.484691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.484552    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 20:15:48.484802 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.484612    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 20:15:48.492330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.492301    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"]
Nov 03 20:15:48.504233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.504205    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-routing-overlay-volume\") pod \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") "
Nov 03 20:15:48.504567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.504531    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-plugins-config-volume\") pod \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") "
Nov 03 20:15:48.505004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.504602    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-routing-overlay-volume" pod "dc97ad46-4774-4efa-a9fd-cb061eaea0d9" (UID: "dc97ad46-4774-4efa-a9fd-cb061eaea0d9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:48.505004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.504649    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-provider-credentials\") pod \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") "
Nov 03 20:15:48.505004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.504710    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-kube-api-access-87clv\" (UniqueName: \"kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-kube-api-access-87clv\") pod \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\" (UID: \"dc97ad46-4774-4efa-a9fd-cb061eaea0d9\") "
Nov 03 20:15:48.505004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.504920    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-plugins-config-volume" pod "dc97ad46-4774-4efa-a9fd-cb061eaea0d9" (UID: "dc97ad46-4774-4efa-a9fd-cb061eaea0d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:48.505917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.505894    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:48.506035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.505921    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:48.507980 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.507952    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-provider-credentials" pod "dc97ad46-4774-4efa-a9fd-cb061eaea0d9" (UID: "dc97ad46-4774-4efa-a9fd-cb061eaea0d9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:48.510391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.510362    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-kube-api-access-87clv" pod "dc97ad46-4774-4efa-a9fd-cb061eaea0d9" (UID: "dc97ad46-4774-4efa-a9fd-cb061eaea0d9"). InnerVolumeSpecName "kube-api-access-87clv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:48.607615 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.607506    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mk2r\" (UniqueName: \"kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-kube-api-access-4mk2r\") pod \"payload-processing-external-model-7988fb85f8-k9qwt\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") " pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.607799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.607646    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-provider-credentials\") pod \"payload-processing-external-model-7988fb85f8-k9qwt\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") " pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.607799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.607737    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-routing-overlay-volume\") pod \"payload-processing-external-model-7988fb85f8-k9qwt\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") " pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.607893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.607862    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-plugins-config-volume\") pod \"payload-processing-external-model-7988fb85f8-k9qwt\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") " pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.608029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.608015    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:48.608075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.608031    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87clv\" (UniqueName: \"kubernetes.io/projected/dc97ad46-4774-4efa-a9fd-cb061eaea0d9-kube-api-access-87clv\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:48.709493 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.709440    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mk2r\" (UniqueName: \"kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-kube-api-access-4mk2r\") pod \"payload-processing-external-model-7988fb85f8-k9qwt\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") " pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.709994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.709524    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-provider-credentials\") pod \"payload-processing-external-model-7988fb85f8-k9qwt\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") " pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.709994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.709559    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-routing-overlay-volume\") pod \"payload-processing-external-model-7988fb85f8-k9qwt\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") " pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.709994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.709645    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-plugins-config-volume\") pod \"payload-processing-external-model-7988fb85f8-k9qwt\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") " pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.710304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.710281    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-routing-overlay-volume\") pod \"payload-processing-external-model-7988fb85f8-k9qwt\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") " pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.710445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.710425    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-plugins-config-volume\") pod \"payload-processing-external-model-7988fb85f8-k9qwt\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") " pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.712078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.712053    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-provider-credentials\") pod \"payload-processing-external-model-7988fb85f8-k9qwt\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") " pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.721838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.721813    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mk2r\" (UniqueName: \"kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-kube-api-access-4mk2r\") pod \"payload-processing-external-model-7988fb85f8-k9qwt\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") " pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.800256 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.800216    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:48.890399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.890366    5659 generic.go:356] "Generic (PLEG): container finished" podID="dc97ad46-4774-4efa-a9fd-cb061eaea0d9" containerID="0acd073c614eb322c4ee43b7063e24fe4fbba35d093769519ef64d2f27c13c7b" exitCode=0
Nov 03 20:15:48.890399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.890391    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"
Nov 03 20:15:48.890691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.890673    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr" event={"ID":"dc97ad46-4774-4efa-a9fd-cb061eaea0d9","Type":"ContainerDied","Data":"0acd073c614eb322c4ee43b7063e24fe4fbba35d093769519ef64d2f27c13c7b"}
Nov 03 20:15:48.890769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.890728    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr" event={"ID":"dc97ad46-4774-4efa-a9fd-cb061eaea0d9","Type":"ContainerDied","Data":"44c58b78bb16ca4f0a59f5a11589ec9a3da7e23d642bec41ebef55274b9eb7c6"}
Nov 03 20:15:48.890769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.890759    5659 scope.go:122] "RemoveContainer" containerID="0acd073c614eb322c4ee43b7063e24fe4fbba35d093769519ef64d2f27c13c7b"
Nov 03 20:15:48.906348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.906324    5659 scope.go:122] "RemoveContainer" containerID="0acd073c614eb322c4ee43b7063e24fe4fbba35d093769519ef64d2f27c13c7b"
Nov 03 20:15:48.906882 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:15:48.906850    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acd073c614eb322c4ee43b7063e24fe4fbba35d093769519ef64d2f27c13c7b\": container with ID starting with 0acd073c614eb322c4ee43b7063e24fe4fbba35d093769519ef64d2f27c13c7b not found: ID does not exist" containerID="0acd073c614eb322c4ee43b7063e24fe4fbba35d093769519ef64d2f27c13c7b"
Nov 03 20:15:48.906978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.906895    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acd073c614eb322c4ee43b7063e24fe4fbba35d093769519ef64d2f27c13c7b"} err="failed to get container status \"0acd073c614eb322c4ee43b7063e24fe4fbba35d093769519ef64d2f27c13c7b\": rpc error: code = NotFound desc = could not find container \"0acd073c614eb322c4ee43b7063e24fe4fbba35d093769519ef64d2f27c13c7b\": container with ID starting with 0acd073c614eb322c4ee43b7063e24fe4fbba35d093769519ef64d2f27c13c7b not found: ID does not exist"
Nov 03 20:15:48.931735 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.931697    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"]
Nov 03 20:15:48.938295 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:48.938262    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-85d8rqbcr"]
Nov 03 20:15:49.095246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:49.095215    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"]
Nov 03 20:15:49.317338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:49.317278    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:49.735581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:49.734063    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"]
Nov 03 20:15:49.827548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:49.827088    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:49.918775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:49.918722    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt" event={"ID":"dc64f67e-f8fb-4bd2-8119-8a2c36513c95","Type":"ContainerStarted","Data":"6a2334b4bb918678ce6cfc7c4b5c0913a2c0e4efbf4893acc2b12d3cbd4da3e6"}
Nov 03 20:15:49.919028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:49.918785    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt" event={"ID":"dc64f67e-f8fb-4bd2-8119-8a2c36513c95","Type":"ContainerStarted","Data":"e5ca8c8f70538426fd151743e3a438e6f2e04689dfc7a8b5a7f5608d2790c477"}
Nov 03 20:15:49.919028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:49.918832    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:15:50.002561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:50.001730    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:50.199505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:50.186243    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt" podStartSLOduration=5.186221167 podStartE2EDuration="5.186221167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:49.950067034 +0000 UTC m=+2302.104969454" watchObservedRunningTime="2026-11-03 20:15:50.186221167 +0000 UTC m=+2302.341123587"
Nov 03 20:15:50.222512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:50.212934    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:50.354561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:50.291563    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc97ad46-4774-4efa-a9fd-cb061eaea0d9" path="/var/lib/kubelet/pods/dc97ad46-4774-4efa-a9fd-cb061eaea0d9/volumes"
Nov 03 20:15:50.538558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:50.477908    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:50.746240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:50.746204    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:50.902864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:50.902769    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:51.113043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:51.112949    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:51.284600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:51.284563    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:51.448935 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:51.448830    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:52.007995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:52.007961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:52.158549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:52.154961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:52.294563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:52.293970    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:52.669424 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:52.669324    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:53.463799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:53.463751    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:53.594853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:53.594809    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:54.127416 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:54.127370    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:54.348503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:54.348043    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:54.738305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:54.737935    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6" podUID="ad9133a5-8589-4761-8c80-4dd93a2195c6" containerName="payload-processing" containerID="cri-o://5bcab7e7fe7f431316a1378cf9aae37c5dbcef4443736cad264be56098c19926" gracePeriod=25
Nov 03 20:15:55.025336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.025292    5659 generic.go:356] "Generic (PLEG): container finished" podID="ad9133a5-8589-4761-8c80-4dd93a2195c6" containerID="5bcab7e7fe7f431316a1378cf9aae37c5dbcef4443736cad264be56098c19926" exitCode=0
Nov 03 20:15:55.025751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.025726    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6" event={"ID":"ad9133a5-8589-4761-8c80-4dd93a2195c6","Type":"ContainerDied","Data":"5bcab7e7fe7f431316a1378cf9aae37c5dbcef4443736cad264be56098c19926"}
Nov 03 20:15:55.180648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.180615    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:15:55.216944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.216898    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-routing-overlay-volume\") pod \"ad9133a5-8589-4761-8c80-4dd93a2195c6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") "
Nov 03 20:15:55.217111 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.217066    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-plugins-config-volume\") pod \"ad9133a5-8589-4761-8c80-4dd93a2195c6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") "
Nov 03 20:15:55.217178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.217158    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-kube-api-access-cbzh6\" (UniqueName: \"kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-kube-api-access-cbzh6\") pod \"ad9133a5-8589-4761-8c80-4dd93a2195c6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") "
Nov 03 20:15:55.217249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.217222    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-provider-credentials\") pod \"ad9133a5-8589-4761-8c80-4dd93a2195c6\" (UID: \"ad9133a5-8589-4761-8c80-4dd93a2195c6\") "
Nov 03 20:15:55.217309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.217269    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-routing-overlay-volume" pod "ad9133a5-8589-4761-8c80-4dd93a2195c6" (UID: "ad9133a5-8589-4761-8c80-4dd93a2195c6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:55.217539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.217516    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-plugins-config-volume" pod "ad9133a5-8589-4761-8c80-4dd93a2195c6" (UID: "ad9133a5-8589-4761-8c80-4dd93a2195c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:55.218062 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.218035    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:55.218153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.218064    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad9133a5-8589-4761-8c80-4dd93a2195c6-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:55.219816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.219794    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-provider-credentials" pod "ad9133a5-8589-4761-8c80-4dd93a2195c6" (UID: "ad9133a5-8589-4761-8c80-4dd93a2195c6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:55.219965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.219875    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-kube-api-access-cbzh6" pod "ad9133a5-8589-4761-8c80-4dd93a2195c6" (UID: "ad9133a5-8589-4761-8c80-4dd93a2195c6"). InnerVolumeSpecName "kube-api-access-cbzh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:55.321435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.321297    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbzh6\" (UniqueName: \"kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-kube-api-access-cbzh6\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:55.321435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.321335    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad9133a5-8589-4761-8c80-4dd93a2195c6-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:55.508839 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:55.508748    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:56.036756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:56.036718    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6" event={"ID":"ad9133a5-8589-4761-8c80-4dd93a2195c6","Type":"ContainerDied","Data":"c7922d50aecaed80a7b996f2f618f59297fda2032c839595cfeaafaee9f6e9be"}
Nov 03 20:15:56.037182 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:56.036777    5659 scope.go:122] "RemoveContainer" containerID="5bcab7e7fe7f431316a1378cf9aae37c5dbcef4443736cad264be56098c19926"
Nov 03 20:15:56.037182 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:56.036731    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"
Nov 03 20:15:56.062881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:56.062850    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"]
Nov 03 20:15:56.067198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:56.067175    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-79c62nkt6"]
Nov 03 20:15:56.249323 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:56.249243    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:56.393102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:56.393064    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:57.147379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:57.147348    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:57.479119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:57.479083    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:57.776925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:57.776840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:57.974501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:57.974450    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad9133a5-8589-4761-8c80-4dd93a2195c6" path="/var/lib/kubelet/pods/ad9133a5-8589-4761-8c80-4dd93a2195c6/volumes"
Nov 03 20:15:58.537606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:58.537568    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:59.181790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:15:59.181756    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:00.030981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:00.030942    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:00.737360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:00.737319    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:00.932219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:00.932182    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:16:01.467812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:01.467779    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:02.159068 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:02.159031    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:02.854028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:02.853987    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:03.602753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:03.602718    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:03.766614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:03.766578    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:03.897254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:03.897174    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:04.359662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:04.359628    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:05.136035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:05.136000    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:05.838993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:05.838959    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:06.047360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:06.047278    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:06.173330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:06.173243    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:06.294980 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:06.294942    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:07.000614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:06.989871    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:07.013600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.011258    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"]
Nov 03 20:16:07.013600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.012531    5659 cpu_manager.go:414] "containerMap: removing container" podUID="ad9133a5-8589-4761-8c80-4dd93a2195c6" containerName="payload-processing"
Nov 03 20:16:07.013600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.012552    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad9133a5-8589-4761-8c80-4dd93a2195c6" containerName="payload-processing"
Nov 03 20:16:07.013600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.012586    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3aa2c645-beca-46e7-a3af-fd1654bdb351" containerName="cleanup"
Nov 03 20:16:07.013600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.012594    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa2c645-beca-46e7-a3af-fd1654bdb351" containerName="cleanup"
Nov 03 20:16:07.013600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.012636    5659 cpu_manager.go:414] "containerMap: removing container" podUID="1601b767-8367-4476-a1b9-c33c58403370" containerName="cleanup"
Nov 03 20:16:07.013600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.012645    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1601b767-8367-4476-a1b9-c33c58403370" containerName="cleanup"
Nov 03 20:16:07.013600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.012839    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="1601b767-8367-4476-a1b9-c33c58403370" containerName="cleanup"
Nov 03 20:16:07.013600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.012862    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad9133a5-8589-4761-8c80-4dd93a2195c6" containerName="payload-processing"
Nov 03 20:16:07.013600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.012877    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa2c645-beca-46e7-a3af-fd1654bdb351" containerName="cleanup"
Nov 03 20:16:07.022128 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.022104    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:07.027805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.027776    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"]
Nov 03 20:16:07.028029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.027825    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"agc-cred-c4c7ed60cafb\""
Nov 03 20:16:07.028291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.028171    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"openshift-service-ca.crt\""
Nov 03 20:16:07.028291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.028192    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"routing-overlay\""
Nov 03 20:16:07.028291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.028257    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-98529d\""
Nov 03 20:16:07.028291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.028266    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"kube-root-ca.crt\""
Nov 03 20:16:07.099988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.099948    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:07.100245 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.100219    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftl66\" (UniqueName: \"kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-kube-api-access-ftl66\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:07.100374 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.100360    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:07.100445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.100389    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:07.140307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.140260    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"]
Nov 03 20:16:07.185671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.185635    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:07.204255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.204211    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:07.204885 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.204852    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftl66\" (UniqueName: \"kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-kube-api-access-ftl66\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:07.205325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.205302    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:07.205470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.205454    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:07.206636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.206614    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:07.206874 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:16:07.206854    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-98529d/agc-cred-c4c7ed60cafb: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:16:07.206981 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:16:07.206970    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:16:07.207178 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:16:07.207165    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-provider-credentials podName:b49d8572-5c18-4ee6-becf-d1fb220e189b nodeName:}" failed. No retries permitted until 2026-11-03 20:16:07.707141696 +0000 UTC m=+2319.862044112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg" (UID: "b49d8572-5c18-4ee6-becf-d1fb220e189b") : secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:16:07.208176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.208154    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:07.221415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.221154    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftl66\" (UniqueName: \"kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-kube-api-access-ftl66\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:07.452562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.451867    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:07.720445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.711652    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"]
Nov 03 20:16:07.720445 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:16:07.712494    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg" podUID="b49d8572-5c18-4ee6-becf-d1fb220e189b"
Nov 03 20:16:07.720445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.718615    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:07.720445 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:16:07.719207    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-98529d/agc-cred-c4c7ed60cafb: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:16:07.720445 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:16:07.719223    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:16:07.720445 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:16:07.719267    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-provider-credentials podName:b49d8572-5c18-4ee6-becf-d1fb220e189b nodeName:}" failed. No retries permitted until 2026-11-03 20:16:08.71925457 +0000 UTC m=+2320.874156980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg" (UID: "b49d8572-5c18-4ee6-becf-d1fb220e189b") : secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:16:07.772325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.772257    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:07.949506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:07.933340    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:08.284154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:08.284110    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:08.293661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:08.293638    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:08.384113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:08.384075    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-kube-api-access-ftl66\" (UniqueName: \"kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-kube-api-access-ftl66\") pod \"b49d8572-5c18-4ee6-becf-d1fb220e189b\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") "
Nov 03 20:16:08.384327 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:08.384225    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-plugins-config-volume\") pod \"b49d8572-5c18-4ee6-becf-d1fb220e189b\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") "
Nov 03 20:16:08.384404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:08.384331    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-routing-overlay-volume\") pod \"b49d8572-5c18-4ee6-becf-d1fb220e189b\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") "
Nov 03 20:16:08.385001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:08.384727    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-plugins-config-volume" pod "b49d8572-5c18-4ee6-becf-d1fb220e189b" (UID: "b49d8572-5c18-4ee6-becf-d1fb220e189b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:08.385001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:08.384769    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-routing-overlay-volume" pod "b49d8572-5c18-4ee6-becf-d1fb220e189b" (UID: "b49d8572-5c18-4ee6-becf-d1fb220e189b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:08.385538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:08.385513    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:08.385646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:08.385543    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b49d8572-5c18-4ee6-becf-d1fb220e189b-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:08.386551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:08.386515    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-kube-api-access-ftl66" pod "b49d8572-5c18-4ee6-becf-d1fb220e189b" (UID: "b49d8572-5c18-4ee6-becf-d1fb220e189b"). InnerVolumeSpecName "kube-api-access-ftl66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:08.487627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:08.487592    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftl66\" (UniqueName: \"kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-kube-api-access-ftl66\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:08.544053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:08.543967    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:08.795909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:08.795805    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg\" (UID: \"b49d8572-5c18-4ee6-becf-d1fb220e189b\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:08.796087 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:16:08.795978    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-98529d/agc-cred-c4c7ed60cafb: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:16:08.796087 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:16:08.795997    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg: secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:16:08.796087 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:16:08.796054    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-provider-credentials podName:b49d8572-5c18-4ee6-becf-d1fb220e189b nodeName:}" failed. No retries permitted until 2026-11-03 20:16:10.796037246 +0000 UTC m=+2322.950939656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg" (UID: "b49d8572-5c18-4ee6-becf-d1fb220e189b") : secret "agc-cred-c4c7ed60cafb" not found
Nov 03 20:16:09.254321 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:09.254286    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:09.303915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:09.303883    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"
Nov 03 20:16:09.406652 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:09.406614    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"]
Nov 03 20:16:09.411173 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:09.411140    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-5b84hbqdg"]
Nov 03 20:16:09.509891 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:09.509804    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b49d8572-5c18-4ee6-becf-d1fb220e189b-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:09.957605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:09.957570    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:10.019913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:10.019874    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49d8572-5c18-4ee6-becf-d1fb220e189b" path="/var/lib/kubelet/pods/b49d8572-5c18-4ee6-becf-d1fb220e189b/volumes"
Nov 03 20:16:10.666203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:10.666102    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:11.331948 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:11.331909    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:12.053798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.053760    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:12.141383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.141330    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8" podUID="760e2a75-4bdd-422d-879c-35d68e2eaa22" containerName="payload-processing" containerID="cri-o://64dc9962e34cd5199006e45369dfb045e92c421f5b62b44dfc59f4ecf7da9288" gracePeriod=25
Nov 03 20:16:12.332167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.332031    5659 generic.go:356] "Generic (PLEG): container finished" podID="760e2a75-4bdd-422d-879c-35d68e2eaa22" containerID="64dc9962e34cd5199006e45369dfb045e92c421f5b62b44dfc59f4ecf7da9288" exitCode=0
Nov 03 20:16:12.332167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.332102    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8" event={"ID":"760e2a75-4bdd-422d-879c-35d68e2eaa22","Type":"ContainerDied","Data":"64dc9962e34cd5199006e45369dfb045e92c421f5b62b44dfc59f4ecf7da9288"}
Nov 03 20:16:12.461132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.461105    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:16:12.572226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.572178    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-routing-overlay-volume\") pod \"760e2a75-4bdd-422d-879c-35d68e2eaa22\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") "
Nov 03 20:16:12.572448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.572264    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-kube-api-access-j2cf6\" (UniqueName: \"kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-kube-api-access-j2cf6\") pod \"760e2a75-4bdd-422d-879c-35d68e2eaa22\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") "
Nov 03 20:16:12.572448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.572309    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-provider-credentials\" (UniqueName: \"kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-provider-credentials\") pod \"760e2a75-4bdd-422d-879c-35d68e2eaa22\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") "
Nov 03 20:16:12.572448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.572416    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-plugins-config-volume\") pod \"760e2a75-4bdd-422d-879c-35d68e2eaa22\" (UID: \"760e2a75-4bdd-422d-879c-35d68e2eaa22\") "
Nov 03 20:16:12.574110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.573906    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-plugins-config-volume" pod "760e2a75-4bdd-422d-879c-35d68e2eaa22" (UID: "760e2a75-4bdd-422d-879c-35d68e2eaa22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:12.574241 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.574087    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-routing-overlay-volume" pod "760e2a75-4bdd-422d-879c-35d68e2eaa22" (UID: "760e2a75-4bdd-422d-879c-35d68e2eaa22"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:12.576261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.576231    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-provider-credentials" pod "760e2a75-4bdd-422d-879c-35d68e2eaa22" (UID: "760e2a75-4bdd-422d-879c-35d68e2eaa22"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:12.576435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.576409    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-kube-api-access-j2cf6" pod "760e2a75-4bdd-422d-879c-35d68e2eaa22" (UID: "760e2a75-4bdd-422d-879c-35d68e2eaa22"). InnerVolumeSpecName "kube-api-access-j2cf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:12.675446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.675402    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:12.675446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.675443    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2cf6\" (UniqueName: \"kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-kube-api-access-j2cf6\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:12.675664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.675459    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/760e2a75-4bdd-422d-879c-35d68e2eaa22-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:12.675664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.675473    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760e2a75-4bdd-422d-879c-35d68e2eaa22-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:12.898262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:12.898074    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:13.341538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:13.341500    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"
Nov 03 20:16:13.342032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:13.341528    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8" event={"ID":"760e2a75-4bdd-422d-879c-35d68e2eaa22","Type":"ContainerDied","Data":"72157b219c10e7ca5d310189fb16e4f9aec7787a37a8fcc140af6f01b7175b63"}
Nov 03 20:16:13.342032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:13.341602    5659 scope.go:122] "RemoveContainer" containerID="64dc9962e34cd5199006e45369dfb045e92c421f5b62b44dfc59f4ecf7da9288"
Nov 03 20:16:13.381028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:13.380961    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"]
Nov 03 20:16:13.386447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:13.386412    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8"]
Nov 03 20:16:13.412664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:13.412622    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c644ljg8" podUID="760e2a75-4bdd-422d-879c-35d68e2eaa22" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.160:50052\" within 1s: context deadline exceeded"
Nov 03 20:16:13.412875 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:16:13.412695    5659 logging.go:55] [core] [Channel #2387 SubChannel #2388]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.160:50052", ServerName: "10.133.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.160:50052: operation was canceled"
Nov 03 20:16:13.644004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:13.643921    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:13.972058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:13.972023    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="760e2a75-4bdd-422d-879c-35d68e2eaa22" path="/var/lib/kubelet/pods/760e2a75-4bdd-422d-879c-35d68e2eaa22/volumes"
Nov 03 20:16:14.764431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:14.764264    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:14.897294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:14.897259    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:15.052236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:15.052145    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:15.396567 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:16:15.394938    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:16:16.335875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:16.335838    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:17.184506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:17.184211    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:18.131681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:18.131643    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:19.033924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:19.033880    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:19.270438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:19.270402    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:19.582173 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:19.582139    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:19.712479 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:19.712438    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:19.940137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:19.939957    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:21.088973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:21.088925    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:21.854876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:21.854835    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:22.333220 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:22.333169    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:22.468399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:22.468370    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:22.630028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:22.629944    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:23.196529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:23.196467    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:23.324189 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:23.324141    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:23.845826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:23.845789    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:24.821736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:24.821695    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:25.541212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:25.541178    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:26.441784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:26.441747    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:27.145448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:27.145405    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:27.343927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:27.343892    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:27.489644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:27.489432    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:27.981810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:27.981774    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:28.343389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:28.343297    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:29.054060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:29.054021    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:29.730279 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:29.730244    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:30.522015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:30.521930    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:32.010150 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:32.010114    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:32.209304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:32.209256    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:33.334990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.334953    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:33.445128 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.445094    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"]
Nov 03 20:16:33.446108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.446081    5659 cpu_manager.go:414] "containerMap: removing container" podUID="760e2a75-4bdd-422d-879c-35d68e2eaa22" containerName="payload-processing"
Nov 03 20:16:33.446108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.446107    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="760e2a75-4bdd-422d-879c-35d68e2eaa22" containerName="payload-processing"
Nov 03 20:16:33.446352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.446307    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="760e2a75-4bdd-422d-879c-35d68e2eaa22" containerName="payload-processing"
Nov 03 20:16:33.452574 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.452550    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.461052 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.461026    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"]
Nov 03 20:16:33.551391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.551350    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wplwn\" (UniqueName: \"kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-kube-api-access-wplwn\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.551590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.551401    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab882b17-b18a-4552-9f94-90deb128bbfd-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.551647 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.551589    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.551704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.551663    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.551750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.551730    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.551855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.551839    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.551922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.551909    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.551966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.551952    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.552114 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.552094    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.653947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.653860    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.653947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.653916    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.654200 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.654015    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.654200 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.654086    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.654200 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.654127    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.654441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.654229    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.654441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.654346    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wplwn\" (UniqueName: \"kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-kube-api-access-wplwn\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.654441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.654363    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.654441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.654379    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab882b17-b18a-4552-9f94-90deb128bbfd-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.654441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.654393    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.654753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.654449    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.654753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.654592    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.654931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.654903    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.655065 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.655045    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab882b17-b18a-4552-9f94-90deb128bbfd-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.656693 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.656670    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.656839 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.656818    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.666703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.666623    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.666838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.666754    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wplwn\" (UniqueName: \"kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-kube-api-access-wplwn\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:33.774569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:33.774530    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:34.073929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:34.073897    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"]
Nov 03 20:16:34.074075 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:16:34.073951    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab882b17_b18a_4552_9f94_90deb128bbfd.slice/crio-3b6cfc9918f1fe5082f71a879bb67ca8678f7b672b4bef4d8e8e019e370bffee WatchSource:0}: Error finding container 3b6cfc9918f1fe5082f71a879bb67ca8678f7b672b4bef4d8e8e019e370bffee: Status 404 returned error can't find the container with id 3b6cfc9918f1fe5082f71a879bb67ca8678f7b672b4bef4d8e8e019e370bffee
Nov 03 20:16:34.077306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:34.077270    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:16:34.077405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:34.077340    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:16:34.077405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:34.077369    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:16:34.158938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:34.154947    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:34.708009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:34.707974    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:34.858506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:34.845266    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:34.879814 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:34.879778    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf" event={"ID":"ab882b17-b18a-4552-9f94-90deb128bbfd","Type":"ContainerStarted","Data":"684873f468de87420ff2f8d60292121f9185ee114d1e08d0369d76d2870b1990"}
Nov 03 20:16:34.879999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:34.879834    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf" event={"ID":"ab882b17-b18a-4552-9f94-90deb128bbfd","Type":"ContainerStarted","Data":"3b6cfc9918f1fe5082f71a879bb67ca8678f7b672b4bef4d8e8e019e370bffee"}
Nov 03 20:16:34.922508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:34.919726    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf" podStartSLOduration=1.919704163 podStartE2EDuration="1.919704163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:34.915991284 +0000 UTC m=+2347.070893704" watchObservedRunningTime="2026-11-03 20:16:34.919704163 +0000 UTC m=+2347.074606584"
Nov 03 20:16:34.993161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:34.991079    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:35.650506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:35.644027    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:35.776618 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:35.776576    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:35.779084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:35.779058    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 20:16:35.779221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:35.779109    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf" podUID="ab882b17-b18a-4552-9f94-90deb128bbfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 20:16:36.515273 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:36.515054    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:36.669308 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:36.669254    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:36.838563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:36.833466    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 20:16:36.838563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:36.833545    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf" podUID="ab882b17-b18a-4552-9f94-90deb128bbfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 20:16:37.775539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:37.775501    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 20:16:37.775733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:37.775571    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf" podUID="ab882b17-b18a-4552-9f94-90deb128bbfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 20:16:37.829665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:37.829630    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:38.627400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:38.627368    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:38.764525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:38.764286    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:38.774975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:38.774937    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 20:16:38.775185 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:38.775010    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf" podUID="ab882b17-b18a-4552-9f94-90deb128bbfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 20:16:39.768405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:39.768351    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:39.787326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:39.787278    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 20:16:39.787669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:39.787635    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf" podUID="ab882b17-b18a-4552-9f94-90deb128bbfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 20:16:40.057149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:40.057059    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:40.204614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:40.204558    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:40.784520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:40.783251    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 20:16:40.784520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:40.783321    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf" podUID="ab882b17-b18a-4552-9f94-90deb128bbfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 20:16:41.093770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:41.093688    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:41.603447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:41.603413    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:41.745640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:41.745601    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:41.780883 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:41.780853    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:41.781414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:41.781383    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:41.782039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:41.782020    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:16:41.930450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:41.930352    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:42.185310 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:42.185182    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:42.340217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:42.340182    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:42.566096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:42.566053    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:42.860275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:42.860197    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:43.048105 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:43.048066    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:43.241244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:43.241195    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:43.398319 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:43.398285    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:43.538190 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:43.538093    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:44.077798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:44.077398    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:44.201619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:44.201582    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:44.347789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:44.347699    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:44.503504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:44.498112    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:44.685129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:44.685042    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:45.510593 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:16:45.510467    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:16:45.573650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:45.557189    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:45.573923 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:45.573884    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:45.574176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:45.574140    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:16:45.610673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:45.610222    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:45.610673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:45.610341    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:45.695939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:45.694679    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 20:16:45.761412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:45.761233    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Nov 03 20:16:45.808317 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:45.806898    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:46.532795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:46.532742    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:47.254186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:47.254148    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:47.952033 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:47.951990    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:48.648434 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:48.648400    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:49.350086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:49.350047    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:50.126716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:50.126681    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:50.929938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:50.929795    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:51.680104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:51.680064    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:52.332419 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:52.332369    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:53.070137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:53.070095    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:53.825458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:53.825421    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:54.580756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:54.580724    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:55.294494 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:55.294444    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"]
Nov 03 20:16:55.345836 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:55.345799    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:56.058307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:56.058264    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:56.943507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:56.942112    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:58.497259 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:58.497225    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:59.369404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:16:59.369320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:00.269507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:00.269063    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:00.298622 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:00.298570    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt" podUID="dc64f67e-f8fb-4bd2-8119-8a2c36513c95" containerName="payload-processing" containerID="cri-o://6a2334b4bb918678ce6cfc7c4b5c0913a2c0e4efbf4893acc2b12d3cbd4da3e6" gracePeriod=25
Nov 03 20:17:01.064852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.064777    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:17:01.092932 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.092893    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:01.129752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.128718    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"]
Nov 03 20:17:01.132398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.130608    5659 cpu_manager.go:414] "containerMap: removing container" podUID="dc64f67e-f8fb-4bd2-8119-8a2c36513c95" containerName="payload-processing"
Nov 03 20:17:01.132398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.130639    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc64f67e-f8fb-4bd2-8119-8a2c36513c95" containerName="payload-processing"
Nov 03 20:17:01.132398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.130829    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc64f67e-f8fb-4bd2-8119-8a2c36513c95" containerName="payload-processing"
Nov 03 20:17:01.166858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.166821    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.172591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.171541    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"]
Nov 03 20:17:01.178954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.178927    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"openshift-service-ca.crt\""
Nov 03 20:17:01.182326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.179955    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"agc-cred-3eafb068aa07\""
Nov 03 20:17:01.182823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.180119    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"payload-processing-external-model-e2e--aa51d60f-dockercfg-wfnqm\""
Nov 03 20:17:01.182961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.180177    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"payload-processing-external-model-plugins-e2e-extmod-a-33e267\""
Nov 03 20:17:01.183046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.180265    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"routing-overlay\""
Nov 03 20:17:01.183119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.180875    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"agc-cred-3be5207307fb\""
Nov 03 20:17:01.183187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.180939    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"kube-root-ca.crt\""
Nov 03 20:17:01.183264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.181959    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-kube-api-access-4mk2r\" (UniqueName: \"kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-kube-api-access-4mk2r\") pod \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") "
Nov 03 20:17:01.183448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.183433    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-routing-overlay-volume\") pod \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") "
Nov 03 20:17:01.184021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.183789    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-routing-overlay-volume" pod "dc64f67e-f8fb-4bd2-8119-8a2c36513c95" (UID: "dc64f67e-f8fb-4bd2-8119-8a2c36513c95"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:01.184232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.184194    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-provider-credentials\") pod \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") "
Nov 03 20:17:01.184902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.184885    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-plugins-config-volume\") pod \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\" (UID: \"dc64f67e-f8fb-4bd2-8119-8a2c36513c95\") "
Nov 03 20:17:01.186790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.186757    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:01.186961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.185583    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-plugins-config-volume" pod "dc64f67e-f8fb-4bd2-8119-8a2c36513c95" (UID: "dc64f67e-f8fb-4bd2-8119-8a2c36513c95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:01.195522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.193430    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-kube-api-access-4mk2r" pod "dc64f67e-f8fb-4bd2-8119-8a2c36513c95" (UID: "dc64f67e-f8fb-4bd2-8119-8a2c36513c95"). InnerVolumeSpecName "kube-api-access-4mk2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:01.195522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.195227    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-provider-credentials" pod "dc64f67e-f8fb-4bd2-8119-8a2c36513c95" (UID: "dc64f67e-f8fb-4bd2-8119-8a2c36513c95"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:01.288779 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.288740    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.289404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.289385    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhdtv\" (UniqueName: \"kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-kube-api-access-xhdtv\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.289688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.289672    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.289863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.289851    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.290068 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.290056    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mk2r\" (UniqueName: \"kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-kube-api-access-4mk2r\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:01.290147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.290139    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:01.290204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.290196    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc64f67e-f8fb-4bd2-8119-8a2c36513c95-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:01.391862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.391764    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhdtv\" (UniqueName: \"kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-kube-api-access-xhdtv\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.392045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.391899    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.392045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.391982    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.392178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.392067    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.392780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.392755    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.392982 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.392956    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.394950 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.394926    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.408538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.408510    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhdtv\" (UniqueName: \"kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-kube-api-access-xhdtv\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.574346 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.574303    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:01.592121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.592086    5659 generic.go:356] "Generic (PLEG): container finished" podID="dc64f67e-f8fb-4bd2-8119-8a2c36513c95" containerID="6a2334b4bb918678ce6cfc7c4b5c0913a2c0e4efbf4893acc2b12d3cbd4da3e6" exitCode=0
Nov 03 20:17:01.592758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.592237    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt" event={"ID":"dc64f67e-f8fb-4bd2-8119-8a2c36513c95","Type":"ContainerDied","Data":"6a2334b4bb918678ce6cfc7c4b5c0913a2c0e4efbf4893acc2b12d3cbd4da3e6"}
Nov 03 20:17:01.592921 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.592905    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt" event={"ID":"dc64f67e-f8fb-4bd2-8119-8a2c36513c95","Type":"ContainerDied","Data":"e5ca8c8f70538426fd151743e3a438e6f2e04689dfc7a8b5a7f5608d2790c477"}
Nov 03 20:17:01.593038 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.593025    5659 scope.go:122] "RemoveContainer" containerID="6a2334b4bb918678ce6cfc7c4b5c0913a2c0e4efbf4893acc2b12d3cbd4da3e6"
Nov 03 20:17:01.593219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.592332    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"
Nov 03 20:17:01.615828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.615795    5659 scope.go:122] "RemoveContainer" containerID="6a2334b4bb918678ce6cfc7c4b5c0913a2c0e4efbf4893acc2b12d3cbd4da3e6"
Nov 03 20:17:01.616236 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:17:01.616212    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a2334b4bb918678ce6cfc7c4b5c0913a2c0e4efbf4893acc2b12d3cbd4da3e6\": container with ID starting with 6a2334b4bb918678ce6cfc7c4b5c0913a2c0e4efbf4893acc2b12d3cbd4da3e6 not found: ID does not exist" containerID="6a2334b4bb918678ce6cfc7c4b5c0913a2c0e4efbf4893acc2b12d3cbd4da3e6"
Nov 03 20:17:01.616341 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.616248    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a2334b4bb918678ce6cfc7c4b5c0913a2c0e4efbf4893acc2b12d3cbd4da3e6"} err="failed to get container status \"6a2334b4bb918678ce6cfc7c4b5c0913a2c0e4efbf4893acc2b12d3cbd4da3e6\": rpc error: code = NotFound desc = could not find container \"6a2334b4bb918678ce6cfc7c4b5c0913a2c0e4efbf4893acc2b12d3cbd4da3e6\": container with ID starting with 6a2334b4bb918678ce6cfc7c4b5c0913a2c0e4efbf4893acc2b12d3cbd4da3e6 not found: ID does not exist"
Nov 03 20:17:01.631239 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.631156    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"]
Nov 03 20:17:01.637259 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.637219    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt"]
Nov 03 20:17:01.876000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.875968    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"]
Nov 03 20:17:01.876127 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:17:01.876019    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode026989c_d59f_4382_b039_bebe81c37b7b.slice/crio-473ef77994759401cfcd84988ffcde88cb58677f797aa2251e0520a89333fb70 WatchSource:0}: Error finding container 473ef77994759401cfcd84988ffcde88cb58677f797aa2251e0520a89333fb70: Status 404 returned error can't find the container with id 473ef77994759401cfcd84988ffcde88cb58677f797aa2251e0520a89333fb70
Nov 03 20:17:01.930468 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.930419    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7988fb85f8-k9qwt" podUID="dc64f67e-f8fb-4bd2-8119-8a2c36513c95" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.172:50052\" within 1s: context deadline exceeded"
Nov 03 20:17:01.930648 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:17:01.930559    5659 logging.go:55] [core] [Channel #2608 SubChannel #2609]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.172:50052", ServerName: "10.133.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.172:50052: operation was canceled"
Nov 03 20:17:01.973918 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:01.973882    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc64f67e-f8fb-4bd2-8119-8a2c36513c95" path="/var/lib/kubelet/pods/dc64f67e-f8fb-4bd2-8119-8a2c36513c95/volumes"
Nov 03 20:17:02.036300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:02.036263    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:02.605425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:02.605386    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk" event={"ID":"e026989c-d59f-4382-b039-bebe81c37b7b","Type":"ContainerStarted","Data":"ce8c7fc643810362034dee469bfbab0a8ca0e12f9524cbef44879570cbbf9434"}
Nov 03 20:17:02.605974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:02.605444    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk" event={"ID":"e026989c-d59f-4382-b039-bebe81c37b7b","Type":"ContainerStarted","Data":"473ef77994759401cfcd84988ffcde88cb58677f797aa2251e0520a89333fb70"}
Nov 03 20:17:02.605974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:02.605518    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:02.628946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:02.628892    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk" podStartSLOduration=17.628877026 podStartE2EDuration="17.628877026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:02.624521782 +0000 UTC m=+2374.779424210" watchObservedRunningTime="2026-11-03 20:17:02.628877026 +0000 UTC m=+2374.783779444"
Nov 03 20:17:02.721649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:02.721621    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:03.459833 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:03.459796    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:04.145227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:04.145188    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:04.656603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:04.656565    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:04.792504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:04.789891    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:05.137659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:05.137628    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:05.743850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:05.741849    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:05.873986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:05.873947    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:06.383503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:06.376688    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:06.537585 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:06.524687    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:07.239207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:07.239146    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:07.894257 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:07.894224    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:08.547644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:08.547612    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:09.254965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:09.254925    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:09.940527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:09.940460    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:10.752087 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:10.752053    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:11.432137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:11.432080    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:12.158183 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:12.158152    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:12.734181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:12.734131    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:12.860233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:12.860190    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:13.100749 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:13.100639    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:13.616075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:13.616042    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:17:13.737027 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:13.736991    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:14.521306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:14.521247    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:15.220831 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:17:15.220785    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:17:15.351093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:15.351061    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:16.030644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:16.030613    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:16.770980 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:16.770933    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:17.454428 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:17.454390    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:18.238944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:18.238904    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:18.998216 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:18.998178    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:19.557757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:19.557715    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:19.798796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:19.798758    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:19.960211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:19.960156    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:20.449713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:20.449623    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:20.605941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:20.605907    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:21.132151 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:21.132121    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:21.991996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:21.991963    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:22.818916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:22.818884    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:23.527818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:23.527776    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:24.254724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:24.254681    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:24.932788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:24.932756    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:25.594048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:25.594014    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:26.264819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:26.264780    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:27.033908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:27.033875    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:27.691015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:27.690983    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:28.363893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:28.363858    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:29.054033 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:29.053996    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:29.730477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:29.730449    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:30.463350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:30.463298    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:31.216405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:31.216168    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:32.066505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:32.055814    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:32.740551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:32.740517    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:33.386906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:33.386865    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:34.180959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:34.180903    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:35.115501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:35.115454    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:36.411788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:36.411742    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:17:36.452460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:36.452424    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:36.461008 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:36.460967    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:36.537632 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:36.537587    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:36.538348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:36.538312    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:36.538842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:36.538818    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:37.517719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:37.517686    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:38.589415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:38.589304    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:38.869568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:38.869057    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:39.332315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:39.332282    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:40.064990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:40.064950    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:40.842539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:40.842434    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:41.507537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:41.507460    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:42.154749 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:42.154717    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:42.841405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:42.841368    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:43.523518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:43.519016    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:44.236214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:44.236167    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:44.990114 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:44.990076    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:45.219599 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:17:45.219562    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:17:45.652916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:45.652872    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:46.364709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:46.364670    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:47.079093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:47.079042    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:47.743254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:47.743197    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:48.435122 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:48.435068    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:49.234222 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:49.234189    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:50.040092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:50.040053    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:50.189690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:50.189654    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:50.725503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:50.725452    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:51.170031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:51.169998    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:51.849758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:51.849713    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:52.517058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:52.517025    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:53.257171 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:53.257132    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:54.045421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:54.045389    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:54.732824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:54.732788    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:55.567764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:55.567728    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:56.429764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:56.429730    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:56.932594 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:56.932558    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:57.085067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:57.085027    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:57.241203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:57.241165    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:58.048554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:58.048519    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:58.771509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:58.771452    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:59.450255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:17:59.450220    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:00.342353 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:00.342316    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:01.240670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:01.240637    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:01.964355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:01.964318    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:02.750739 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:02.750703    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:03.510493 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:03.510451    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:03.858358 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:03.858256    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:03.984807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:03.984769    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:04.340080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:04.340044    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:04.764204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:04.764168    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:04.905460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:04.905409    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:05.945099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:05.945063    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:06.645631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:06.645571    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:07.340152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:07.340109    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:08.124661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:08.124624    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:09.057466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:09.057380    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:09.739102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:09.739056    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:10.416753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:10.416719    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:11.221690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:11.221650    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:11.938039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:11.937995    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:12.627824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:12.627788    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:13.352362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:13.352315    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:14.143916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:14.143866    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:14.829250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:14.829207    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:15.219120 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:18:15.219087    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:18:15.620907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:15.620822    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:16.517171 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:16.517103    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:17.268697 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:17.268622    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:17.957886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:17.957842    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:18.818476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:18.818447    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:19.459140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:19.459101    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:20.175283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:20.175249    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:21.008772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:21.008739    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:21.664135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:21.664105    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:22.458355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:22.458312    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:23.228800 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:23.228767    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:24.030817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:24.030777    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:24.848328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:24.848291    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:25.642378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:25.642332    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:26.654093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:26.654059    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:27.420056 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:27.420023    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:28.725388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:28.721802    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:29.778804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:29.778738    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:30.268838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:30.268808    5659 scope.go:122] "RemoveContainer" containerID="14973618a0a8a08600408b85d60d4287e922b2d797715e3bcf267fd93d98ac32"
Nov 03 20:18:30.557009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:30.556902    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:31.265508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:31.265393    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:32.067588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:32.063494    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:32.764317 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:32.764281    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:33.431785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:33.431753    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:34.084868 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:34.084833    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:34.807134 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:34.807101    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:35.499295 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:35.499266    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:35.641504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:35.634121    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:36.226963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:36.226926    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:36.960680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:36.960647    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:37.732138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:37.732098    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:38.422140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:38.422107    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:38.928331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:38.928295    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:39.138123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:39.138082    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:39.895520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:39.895474    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:40.661222 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:40.661179    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:41.478410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:41.478366    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:42.182173 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:42.182121    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:42.975340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:42.975301    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:43.943969 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:43.943932    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:45.227293 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:18:45.227253    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:18:45.280586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:45.280474    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:46.281936 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:46.281899    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:47.775875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:47.775838    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:48.943746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:48.943671    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:49.839883 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:49.839850    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:50.153646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:50.153548    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:50.298516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:50.295573    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:50.884656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:50.884572    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:51.602506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:51.589370    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:52.519911 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:52.519275    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:52.826691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:52.826601    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:53.304845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:53.304808    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:54.155562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:54.155524    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:55.016778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:55.016730    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:55.693527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:55.693494    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:56.435132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:56.435090    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:57.114960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:57.114932    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:57.850502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:57.850452    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:58.528938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:58.528901    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:59.275875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:18:59.275817    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:00.151664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:00.151630    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:01.260294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:01.260257    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:01.740603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:01.740565    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:01.892109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:01.892057    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:02.841074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:02.841034    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:03.729073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:03.729017    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:03.878389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:03.878346    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:04.958141 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:04.958101    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:05.090837 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:05.090807    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:05.229617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:05.228274    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:05.983915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:05.983820    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:06.104833 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:06.104785    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:06.849850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:06.849814    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:07.340740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:07.340697    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:07.559276 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:07.542681    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:08.342313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:08.342271    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:08.682910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:08.682812    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:08.812047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:08.812011    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:08.971993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:08.971952    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:09.367390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:09.366771    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:10.258351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:10.258315    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:11.074530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:11.074451    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:11.887551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:11.887515    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:12.616928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:12.616888    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:13.329401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:13.329339    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:14.127388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:14.127356    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:14.829538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:14.829502    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:15.218619 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:19:15.218585    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:19:15.604559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:15.603966    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:16.451453 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:16.451416    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:17.231715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:17.231677    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:17.917275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:17.917241    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:18.655060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:18.655020    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:19.274952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:19.274917    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:19.478645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:19.478612    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:19.859015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:19.858978    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:19.993242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:19.993212    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:20.919535 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:20.915842    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:21.152621 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:21.151048    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:21.487524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:21.407046    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:19:21.487524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:21.423909    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:19:21.539154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:21.539112    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:21.541966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:21.541048    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:21.541966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:21.541356    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:19:21.591355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:21.591242    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:21.684028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:21.683363    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:21.804530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:21.803028    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:21.923123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:21.923077    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:22.373566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:22.370537    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:22.547376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:22.547325    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:22.550727 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:22.549734    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Nov 03 20:19:22.564510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:22.553558    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:19:22.564510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:22.553665    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:19:22.656078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:22.653922    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:19:22.656847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:22.656824    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:19:23.162948 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:23.162907    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:23.929258 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:23.929223    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:24.590242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:24.590212    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:25.263923 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:25.263882    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:25.948901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:25.948864    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:26.655759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:26.655730    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:27.356358 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:27.356320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:27.633897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:27.633800    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"]
Nov 03 20:19:27.634106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:27.634080    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6" podUID="033f77a2-caed-43b4-8c63-30248e679ea2" containerName="istio-proxy" containerID="cri-o://8d020de094144ab7f0abf14cee2895b13fe7608593f314d1471f5d86e5d77f35" gracePeriod=30
Nov 03 20:19:28.121853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:28.121818    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:28.820051 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:28.820020    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:29.734662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:29.734622    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:30.467794 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:30.467755    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:31.115362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:31.115328    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:31.833338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:31.833301    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:32.717862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:32.717830    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:33.002672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.002639    5659 generic.go:356] "Generic (PLEG): container finished" podID="033f77a2-caed-43b4-8c63-30248e679ea2" containerID="8d020de094144ab7f0abf14cee2895b13fe7608593f314d1471f5d86e5d77f35" exitCode=0
Nov 03 20:19:33.003149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.002719    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6" event={"ID":"033f77a2-caed-43b4-8c63-30248e679ea2","Type":"ContainerDied","Data":"8d020de094144ab7f0abf14cee2895b13fe7608593f314d1471f5d86e5d77f35"}
Nov 03 20:19:33.003149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.002770    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6" event={"ID":"033f77a2-caed-43b4-8c63-30248e679ea2","Type":"ContainerDied","Data":"b53a819b2741f200c8874d296e7fd711df468647cbc86a7ada08f6f729d573ad"}
Nov 03 20:19:33.003149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.002787    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b53a819b2741f200c8874d296e7fd711df468647cbc86a7ada08f6f729d573ad"
Nov 03 20:19:33.013426 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.013400    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:19:33.055119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.055083    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"]
Nov 03 20:19:33.056426 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.056394    5659 cpu_manager.go:414] "containerMap: removing container" podUID="033f77a2-caed-43b4-8c63-30248e679ea2" containerName="istio-proxy"
Nov 03 20:19:33.056426 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.056424    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033f77a2-caed-43b4-8c63-30248e679ea2" containerName="istio-proxy"
Nov 03 20:19:33.056665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.056625    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="033f77a2-caed-43b4-8c63-30248e679ea2" containerName="istio-proxy"
Nov 03 20:19:33.061073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.061045    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.061227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.061165    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"]
Nov 03 20:19:33.069844 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.069783    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.075012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.070635    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-envoy\") pod \"033f77a2-caed-43b4-8c63-30248e679ea2\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") "
Nov 03 20:19:33.075012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.070689    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-kube-api-access-dr6mt\" (UniqueName: \"kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-kube-api-access-dr6mt\") pod \"033f77a2-caed-43b4-8c63-30248e679ea2\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") "
Nov 03 20:19:33.075012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.070774    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-credential-socket\") pod \"033f77a2-caed-43b4-8c63-30248e679ea2\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") "
Nov 03 20:19:33.075012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.070818    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033f77a2-caed-43b4-8c63-30248e679ea2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033f77a2-caed-43b4-8c63-30248e679ea2-istiod-ca-cert\") pod \"033f77a2-caed-43b4-8c63-30248e679ea2\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") "
Nov 03 20:19:33.075012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.070952    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-socket\") pod \"033f77a2-caed-43b4-8c63-30248e679ea2\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") "
Nov 03 20:19:33.075012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.071010    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-istio-token\" (UniqueName: \"kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-istio-token\") pod \"033f77a2-caed-43b4-8c63-30248e679ea2\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") "
Nov 03 20:19:33.075012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.071134    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/033f77a2-caed-43b4-8c63-30248e679ea2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033f77a2-caed-43b4-8c63-30248e679ea2-istio-podinfo\") pod \"033f77a2-caed-43b4-8c63-30248e679ea2\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") "
Nov 03 20:19:33.075012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.071161    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-data\") pod \"033f77a2-caed-43b4-8c63-30248e679ea2\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") "
Nov 03 20:19:33.075012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.071187    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-certs\") pod \"033f77a2-caed-43b4-8c63-30248e679ea2\" (UID: \"033f77a2-caed-43b4-8c63-30248e679ea2\") "
Nov 03 20:19:33.075012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.073687    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-data" pod "033f77a2-caed-43b4-8c63-30248e679ea2" (UID: "033f77a2-caed-43b4-8c63-30248e679ea2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:33.075012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.074039    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-socket" pod "033f77a2-caed-43b4-8c63-30248e679ea2" (UID: "033f77a2-caed-43b4-8c63-30248e679ea2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:33.075012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.074262    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-credential-socket" pod "033f77a2-caed-43b4-8c63-30248e679ea2" (UID: "033f77a2-caed-43b4-8c63-30248e679ea2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:33.075012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.074623    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033f77a2-caed-43b4-8c63-30248e679ea2-istiod-ca-cert" pod "033f77a2-caed-43b4-8c63-30248e679ea2" (UID: "033f77a2-caed-43b4-8c63-30248e679ea2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:33.075966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.075936    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-certs" pod "033f77a2-caed-43b4-8c63-30248e679ea2" (UID: "033f77a2-caed-43b4-8c63-30248e679ea2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:33.080306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.080279    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"]
Nov 03 20:19:33.080525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.080504    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-kube-api-access-dr6mt" pod "033f77a2-caed-43b4-8c63-30248e679ea2" (UID: "033f77a2-caed-43b4-8c63-30248e679ea2"). InnerVolumeSpecName "kube-api-access-dr6mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:33.080633 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.080539    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-istio-token" pod "033f77a2-caed-43b4-8c63-30248e679ea2" (UID: "033f77a2-caed-43b4-8c63-30248e679ea2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:33.080633 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.080599    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/033f77a2-caed-43b4-8c63-30248e679ea2-istio-podinfo" pod "033f77a2-caed-43b4-8c63-30248e679ea2" (UID: "033f77a2-caed-43b4-8c63-30248e679ea2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:33.081884 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.080998    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 20:19:33.081884 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.081655    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ee9d78131bed\""
Nov 03 20:19:33.081884 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.081816    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 20:19:33.081884 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.081840    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-51f23d5569d8\""
Nov 03 20:19:33.082187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.081909    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 20:19:33.082431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.082271    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"agc-cred-70adf96ae88c\""
Nov 03 20:19:33.082431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.082341    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"openshift-service-ca.crt\""
Nov 03 20:19:33.082431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.082403    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"kube-root-ca.crt\""
Nov 03 20:19:33.082762 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.082740    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 20:19:33.083227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.082917    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5lpr4\""
Nov 03 20:19:33.083290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.082976    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"routing-overlay\""
Nov 03 20:19:33.083290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.083037    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"payload-processing-external-model-e2e--c4b4e61e-dockercfg-p7bnm\""
Nov 03 20:19:33.083290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.083184    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-98529d\""
Nov 03 20:19:33.093210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.093185    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"]
Nov 03 20:19:33.093624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.093604    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-envoy" pod "033f77a2-caed-43b4-8c63-30248e679ea2" (UID: "033f77a2-caed-43b4-8c63-30248e679ea2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:33.174179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.174141    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials\") pod \"payload-processing-external-model-588d8bd94c-4cxs6\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") " pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.174354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.174255    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.174354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.174291    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26n72\" (UniqueName: \"kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-kube-api-access-26n72\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.174432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.174369    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-routing-overlay-volume\") pod \"payload-processing-external-model-588d8bd94c-4cxs6\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") " pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.174548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.174520    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.174698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.174682    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.174736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.174722    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hscq\" (UniqueName: \"kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-kube-api-access-5hscq\") pod \"payload-processing-external-model-588d8bd94c-4cxs6\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") " pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.174872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.174853    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-plugins-config-volume\") pod \"payload-processing-external-model-588d8bd94c-4cxs6\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") " pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.175119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.175098    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:33.175283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.175122    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/033f77a2-caed-43b4-8c63-30248e679ea2-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:33.175283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.175137    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:33.175283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.175153    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:33.175283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.175168    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:33.175283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.175184    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr6mt\" (UniqueName: \"kubernetes.io/projected/033f77a2-caed-43b4-8c63-30248e679ea2-kube-api-access-dr6mt\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:33.175283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.175199    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:33.175283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.175212    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/033f77a2-caed-43b4-8c63-30248e679ea2-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:33.175283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.175227    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/033f77a2-caed-43b4-8c63-30248e679ea2-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:33.276462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.276359    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.276462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.276413    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26n72\" (UniqueName: \"kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-kube-api-access-26n72\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.276695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.276590    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-routing-overlay-volume\") pod \"payload-processing-external-model-588d8bd94c-4cxs6\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") " pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.276749 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.276705    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.276862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.276843    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.276913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.276891    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hscq\" (UniqueName: \"kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-kube-api-access-5hscq\") pod \"payload-processing-external-model-588d8bd94c-4cxs6\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") " pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.277000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.276983    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-plugins-config-volume\") pod \"payload-processing-external-model-588d8bd94c-4cxs6\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") " pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.277605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.277227    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials\") pod \"payload-processing-external-model-588d8bd94c-4cxs6\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") " pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.277605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.277531    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-routing-overlay-volume\") pod \"payload-processing-external-model-588d8bd94c-4cxs6\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") " pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.277847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.277821    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.278091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.278026    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-plugins-config-volume\") pod \"payload-processing-external-model-588d8bd94c-4cxs6\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") " pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.278166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.278087    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.279649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.279623    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.279770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.279753    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials\") pod \"payload-processing-external-model-588d8bd94c-4cxs6\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") " pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.288702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.288682    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26n72\" (UniqueName: \"kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-kube-api-access-26n72\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.288878 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.288861    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hscq\" (UniqueName: \"kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-kube-api-access-5hscq\") pod \"payload-processing-external-model-588d8bd94c-4cxs6\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") " pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.428358 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.428321    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:33.441772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.441732    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:33.461652 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.461619    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:33.843555 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:19:33.843516    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67208b39_1303_4730_abc7_5e86f2547dc8.slice/crio-bdc19f736dda12cef5186cb5d81f11c979d27418d28eda036f77b5ef6d44920b WatchSource:0}: Error finding container bdc19f736dda12cef5186cb5d81f11c979d27418d28eda036f77b5ef6d44920b: Status 404 returned error can't find the container with id bdc19f736dda12cef5186cb5d81f11c979d27418d28eda036f77b5ef6d44920b
Nov 03 20:19:33.843724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.843660    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"]
Nov 03 20:19:33.852666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:33.852640    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"]
Nov 03 20:19:34.015754 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:34.015716    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg" event={"ID":"e4500a62-d02c-4231-9774-44c4edcf44e6","Type":"ContainerStarted","Data":"eb3ba78dc37dbe7c4424acde14adddd88dcc7c2256c40d77b4eb2957a2854e25"}
Nov 03 20:19:34.016428 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:34.016402    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg" event={"ID":"e4500a62-d02c-4231-9774-44c4edcf44e6","Type":"ContainerStarted","Data":"a9e71c9aa6264f41b60289d982b2c7bf1103a66b9643f60240beb4921eb37491"}
Nov 03 20:19:34.017267 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:34.017245    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:34.019113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:34.019086    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"
Nov 03 20:19:34.020116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:34.020091    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6" event={"ID":"67208b39-1303-4730-abc7-5e86f2547dc8","Type":"ContainerStarted","Data":"c1a49fdb950b76c8a580e765dbd424cff93f693aac607d6d2ce6bd39ec35c3b5"}
Nov 03 20:19:34.020851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:34.020827    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6" event={"ID":"67208b39-1303-4730-abc7-5e86f2547dc8","Type":"ContainerStarted","Data":"bdc19f736dda12cef5186cb5d81f11c979d27418d28eda036f77b5ef6d44920b"}
Nov 03 20:19:34.021726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:34.021703    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:34.076755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:34.076704    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg" podStartSLOduration=13.076691955 podStartE2EDuration="13.076691955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:34.073598946 +0000 UTC m=+2526.228501365" watchObservedRunningTime="2026-11-03 20:19:34.076691955 +0000 UTC m=+2526.231594391"
Nov 03 20:19:34.091001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:34.090964    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"]
Nov 03 20:19:34.098655 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:34.098627    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-dllj6"]
Nov 03 20:19:34.117871 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:34.117824    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6" podStartSLOduration=27.117812289 podStartE2EDuration="27.117812289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:34.113395371 +0000 UTC m=+2526.268297818" watchObservedRunningTime="2026-11-03 20:19:34.117812289 +0000 UTC m=+2526.272714709"
Nov 03 20:19:34.188249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:34.188172    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:34.845400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:34.845364    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:35.554843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:35.554807    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:35.975116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:35.975079    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033f77a2-caed-43b4-8c63-30248e679ea2" path="/var/lib/kubelet/pods/033f77a2-caed-43b4-8c63-30248e679ea2/volumes"
Nov 03 20:19:36.253681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:36.253595    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:36.961640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:36.961610    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:37.653598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:37.653535    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:38.341882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:38.341846    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:38.985053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:38.984969    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:39.131400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:39.131359    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:39.758895 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:39.758855    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:40.446337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:40.446303    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:41.127294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:41.127262    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:41.925950 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:41.925914    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:42.950713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:42.950628    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:43.951910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:43.951871    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:45.038687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:45.038651    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:19:45.075195 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:45.075161    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:45.296579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:45.291641    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:45.327798 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:19:45.327736    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:19:45.427198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:45.427141    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:45.942131 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:45.941628    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnfsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:19:45.942131 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:45.941699    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" podUID="9fa9e0c2-1bfa-4b41-b033-8d286b4c63be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:46.044981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:46.044953    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:19:46.366359 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:46.366323    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:47.337837 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:47.337804    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:48.051981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:48.051946    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:48.979666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:48.979623    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:49.723543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:49.723511    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:50.452520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:50.452477    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:51.241800 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:51.241765    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:51.931705 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:51.931674    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:52.729891 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:52.729858    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:53.514235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:53.514202    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:54.239439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:54.239408    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:54.950932 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:54.950894    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:55.584946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:55.584909    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:56.260828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:56.260781    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:57.061024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:57.060991    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:57.735159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:57.735120    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:58.474338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:58.474294    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:58.598275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:58.598241    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"]
Nov 03 20:19:58.598449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:58.598426    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" podUID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" containerName="istio-proxy" containerID="cri-o://0404f47c6bca8766762ed51e6ab6b0976667dcf1bca9de951952290a5bb97e32" gracePeriod=30
Nov 03 20:19:59.171030 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:59.170995    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:59.857933 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:19:59.857895    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:00.628832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:00.628794    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:01.313551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:01.313518    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:02.047205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:02.047172    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:02.785846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:02.785817    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:02.977175 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:02.977142    5659 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:20:02.977354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:02.977228    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" podUID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:20:03.457866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.457832    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:03.914890 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.914864    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:20:03.962955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.962921    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-kube-api-access-lqslh\" (UniqueName: \"kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-kube-api-access-lqslh\") pod \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") "
Nov 03 20:20:03.963167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.962996    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-certs\") pod \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") "
Nov 03 20:20:03.963167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.963133    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istiod-ca-cert\") pod \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") "
Nov 03 20:20:03.963392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.963358    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-certs" pod "c6a091b5-503d-4f6a-8864-a02283bfb3c1" (UID: "c6a091b5-503d-4f6a-8864-a02283bfb3c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:03.963560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.963529    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istiod-ca-cert" pod "c6a091b5-503d-4f6a-8864-a02283bfb3c1" (UID: "c6a091b5-503d-4f6a-8864-a02283bfb3c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:03.963702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.963570    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-envoy\") pod \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") "
Nov 03 20:20:03.963702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.963635    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-data\") pod \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") "
Nov 03 20:20:03.963912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.963895    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-data" pod "c6a091b5-503d-4f6a-8864-a02283bfb3c1" (UID: "c6a091b5-503d-4f6a-8864-a02283bfb3c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:03.963977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.963935    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-credential-socket\") pod \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") "
Nov 03 20:20:03.964043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.964017    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-token\" (UniqueName: \"kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-token\") pod \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") "
Nov 03 20:20:03.964539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.964164    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-credential-socket" pod "c6a091b5-503d-4f6a-8864-a02283bfb3c1" (UID: "c6a091b5-503d-4f6a-8864-a02283bfb3c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:03.964539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.964186    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-socket\") pod \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") "
Nov 03 20:20:03.964539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.964244    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-podinfo\") pod \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\" (UID: \"c6a091b5-503d-4f6a-8864-a02283bfb3c1\") "
Nov 03 20:20:03.964539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.964432    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-socket" pod "c6a091b5-503d-4f6a-8864-a02283bfb3c1" (UID: "c6a091b5-503d-4f6a-8864-a02283bfb3c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:03.965197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.965072    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:03.965197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.965100    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:03.965197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.965115    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:03.965197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.965130    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:03.965197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.965143    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:03.967123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.967093    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-envoy" pod "c6a091b5-503d-4f6a-8864-a02283bfb3c1" (UID: "c6a091b5-503d-4f6a-8864-a02283bfb3c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:03.968368 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.968300    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-kube-api-access-lqslh" pod "c6a091b5-503d-4f6a-8864-a02283bfb3c1" (UID: "c6a091b5-503d-4f6a-8864-a02283bfb3c1"). InnerVolumeSpecName "kube-api-access-lqslh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:03.978172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.978125    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-podinfo" pod "c6a091b5-503d-4f6a-8864-a02283bfb3c1" (UID: "c6a091b5-503d-4f6a-8864-a02283bfb3c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:03.995855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:03.995697    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-token" pod "c6a091b5-503d-4f6a-8864-a02283bfb3c1" (UID: "c6a091b5-503d-4f6a-8864-a02283bfb3c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:04.067694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.067652    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:04.067694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.067691    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqslh\" (UniqueName: \"kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-kube-api-access-lqslh\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:04.067954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.067707    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:04.067954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.067721    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a091b5-503d-4f6a-8864-a02283bfb3c1-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:04.285559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.285142    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:04.526955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.526919    5659 generic.go:356] "Generic (PLEG): container finished" podID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" containerID="0404f47c6bca8766762ed51e6ab6b0976667dcf1bca9de951952290a5bb97e32" exitCode=0
Nov 03 20:20:04.527422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.526969    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"
Nov 03 20:20:04.527422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.526986    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" event={"ID":"c6a091b5-503d-4f6a-8864-a02283bfb3c1","Type":"ContainerDied","Data":"0404f47c6bca8766762ed51e6ab6b0976667dcf1bca9de951952290a5bb97e32"}
Nov 03 20:20:04.527422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.527030    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69" event={"ID":"c6a091b5-503d-4f6a-8864-a02283bfb3c1","Type":"ContainerDied","Data":"5c03e046e91be0d8b027c44cc752b3f1ad9ff868769161cd9f7f0d1e721adf1b"}
Nov 03 20:20:04.527422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.527050    5659 scope.go:122] "RemoveContainer" containerID="0404f47c6bca8766762ed51e6ab6b0976667dcf1bca9de951952290a5bb97e32"
Nov 03 20:20:04.541994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.541970    5659 scope.go:122] "RemoveContainer" containerID="0404f47c6bca8766762ed51e6ab6b0976667dcf1bca9de951952290a5bb97e32"
Nov 03 20:20:04.543567 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:04.543531    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0404f47c6bca8766762ed51e6ab6b0976667dcf1bca9de951952290a5bb97e32\": container with ID starting with 0404f47c6bca8766762ed51e6ab6b0976667dcf1bca9de951952290a5bb97e32 not found: ID does not exist" containerID="0404f47c6bca8766762ed51e6ab6b0976667dcf1bca9de951952290a5bb97e32"
Nov 03 20:20:04.543718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.543579    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0404f47c6bca8766762ed51e6ab6b0976667dcf1bca9de951952290a5bb97e32"} err="failed to get container status \"0404f47c6bca8766762ed51e6ab6b0976667dcf1bca9de951952290a5bb97e32\": rpc error: code = NotFound desc = could not find container \"0404f47c6bca8766762ed51e6ab6b0976667dcf1bca9de951952290a5bb97e32\": container with ID starting with 0404f47c6bca8766762ed51e6ab6b0976667dcf1bca9de951952290a5bb97e32 not found: ID does not exist"
Nov 03 20:20:04.565125 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.565090    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"]
Nov 03 20:20:04.569466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:04.569431    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-d5n69"]
Nov 03 20:20:05.004314 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:05.004279    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:05.745510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:05.740061    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:06.015822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:06.013284    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" path="/var/lib/kubelet/pods/c6a091b5-503d-4f6a-8864-a02283bfb3c1/volumes"
Nov 03 20:20:06.524034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:06.523992    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:07.350232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:07.350146    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:08.122430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:08.122393    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:08.770750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:08.770716    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:09.529931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:09.529869    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:10.279477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.279444    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:10.420974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.420939    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:10.506513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.483282    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"]
Nov 03 20:20:10.531569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.508109    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" containerName="istio-proxy"
Nov 03 20:20:10.531819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.531789    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" containerName="istio-proxy"
Nov 03 20:20:10.532256 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.532232    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a091b5-503d-4f6a-8864-a02283bfb3c1" containerName="istio-proxy"
Nov 03 20:20:10.538762 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.538734    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"]
Nov 03 20:20:10.542823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.539073    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:10.590531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.588981    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:10.590531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.589223    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:10.590531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.589263    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vc7\" (UniqueName: \"kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-kube-api-access-k7vc7\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:10.590531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.589303    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:10.690724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.690682    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vc7\" (UniqueName: \"kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-kube-api-access-k7vc7\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:10.690724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.690727    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:10.690993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.690883    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:10.691141 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.691120    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:10.691585 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.691553    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:10.691734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.691711    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:10.693600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.693574    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:10.704265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.704245    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vc7\" (UniqueName: \"kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-kube-api-access-k7vc7\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:10.918178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:10.918079    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:11.145615 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:11.145581    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:11.318606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:11.318565    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"]
Nov 03 20:20:11.323143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:11.323110    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"]
Nov 03 20:20:11.330564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:11.329294    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"]
Nov 03 20:20:11.332161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:11.331321    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:11.456761 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:11.456726    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:12.009504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:12.004339    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:12.034507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:12.032813    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:12.034507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:12.032926    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz" event={"ID":"4ef30fea-df01-4a8e-a60d-d86443df5a57","Type":"ContainerStarted","Data":"18c028a4f2b485ca812babad7c6869dd340ff6ff0e0f918a2e55b8eedaaf0177"}
Nov 03 20:20:12.034507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:12.032954    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz" event={"ID":"4ef30fea-df01-4a8e-a60d-d86443df5a57","Type":"ContainerStarted","Data":"5a692533abc147cbf16fb01c9023383f0a535b47d189a8096b7ce16c88b3dbb7"}
Nov 03 20:20:12.048664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:12.048611    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz" podStartSLOduration=2.048596952 podStartE2EDuration="2.048596952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:12.047004067 +0000 UTC m=+2564.201906486" watchObservedRunningTime="2026-11-03 20:20:12.048596952 +0000 UTC m=+2564.203499370"
Nov 03 20:20:12.468564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:12.451017    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:13.288184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:13.288148    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:13.939863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:13.939831    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:14.651879 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:14.651843    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:15.219176 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:15.219143    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:20:15.332851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:15.332812    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:16.045261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.045226    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:16.324284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.324195    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk" podUID="e026989c-d59f-4382-b039-bebe81c37b7b" containerName="payload-processing" containerID="cri-o://ce8c7fc643810362034dee469bfbab0a8ca0e12f9524cbef44879570cbbf9434" gracePeriod=25
Nov 03 20:20:16.620606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.620574    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:20:16.657999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.657962    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-provider-credentials\") pod \"e026989c-d59f-4382-b039-bebe81c37b7b\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") "
Nov 03 20:20:16.658360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.658334    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-plugins-config-volume\") pod \"e026989c-d59f-4382-b039-bebe81c37b7b\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") "
Nov 03 20:20:16.659953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.659915    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-plugins-config-volume" pod "e026989c-d59f-4382-b039-bebe81c37b7b" (UID: "e026989c-d59f-4382-b039-bebe81c37b7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:16.661088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.661065    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-provider-credentials" pod "e026989c-d59f-4382-b039-bebe81c37b7b" (UID: "e026989c-d59f-4382-b039-bebe81c37b7b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:16.761090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.761042    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-kube-api-access-xhdtv\" (UniqueName: \"kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-kube-api-access-xhdtv\") pod \"e026989c-d59f-4382-b039-bebe81c37b7b\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") "
Nov 03 20:20:16.761371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.761352    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-routing-overlay-volume\") pod \"e026989c-d59f-4382-b039-bebe81c37b7b\" (UID: \"e026989c-d59f-4382-b039-bebe81c37b7b\") "
Nov 03 20:20:16.761795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.761762    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-routing-overlay-volume" pod "e026989c-d59f-4382-b039-bebe81c37b7b" (UID: "e026989c-d59f-4382-b039-bebe81c37b7b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:16.762170 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.762149    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:16.762265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.762174    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:16.762265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.762191    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e026989c-d59f-4382-b039-bebe81c37b7b-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:16.763352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.763320    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-kube-api-access-xhdtv" pod "e026989c-d59f-4382-b039-bebe81c37b7b" (UID: "e026989c-d59f-4382-b039-bebe81c37b7b"). InnerVolumeSpecName "kube-api-access-xhdtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:16.797151 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.797108    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:16.864272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:16.864170    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhdtv\" (UniqueName: \"kubernetes.io/projected/e026989c-d59f-4382-b039-bebe81c37b7b-kube-api-access-xhdtv\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:17.026934 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.026890    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz" podUID="4ef30fea-df01-4a8e-a60d-d86443df5a57" containerName="payload-processing" containerID="cri-o://18c028a4f2b485ca812babad7c6869dd340ff6ff0e0f918a2e55b8eedaaf0177" gracePeriod=25
Nov 03 20:20:17.119991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.119911    5659 generic.go:356] "Generic (PLEG): container finished" podID="e026989c-d59f-4382-b039-bebe81c37b7b" containerID="ce8c7fc643810362034dee469bfbab0a8ca0e12f9524cbef44879570cbbf9434" exitCode=0
Nov 03 20:20:17.119991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.119957    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"
Nov 03 20:20:17.120181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.119996    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk" event={"ID":"e026989c-d59f-4382-b039-bebe81c37b7b","Type":"ContainerDied","Data":"ce8c7fc643810362034dee469bfbab0a8ca0e12f9524cbef44879570cbbf9434"}
Nov 03 20:20:17.120181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.120048    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk" event={"ID":"e026989c-d59f-4382-b039-bebe81c37b7b","Type":"ContainerDied","Data":"473ef77994759401cfcd84988ffcde88cb58677f797aa2251e0520a89333fb70"}
Nov 03 20:20:17.120181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.120070    5659 scope.go:122] "RemoveContainer" containerID="ce8c7fc643810362034dee469bfbab0a8ca0e12f9524cbef44879570cbbf9434"
Nov 03 20:20:17.143681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.143646    5659 scope.go:122] "RemoveContainer" containerID="ce8c7fc643810362034dee469bfbab0a8ca0e12f9524cbef44879570cbbf9434"
Nov 03 20:20:17.144036 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:17.144012    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8c7fc643810362034dee469bfbab0a8ca0e12f9524cbef44879570cbbf9434\": container with ID starting with ce8c7fc643810362034dee469bfbab0a8ca0e12f9524cbef44879570cbbf9434 not found: ID does not exist" containerID="ce8c7fc643810362034dee469bfbab0a8ca0e12f9524cbef44879570cbbf9434"
Nov 03 20:20:17.144124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.144050    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8c7fc643810362034dee469bfbab0a8ca0e12f9524cbef44879570cbbf9434"} err="failed to get container status \"ce8c7fc643810362034dee469bfbab0a8ca0e12f9524cbef44879570cbbf9434\": rpc error: code = NotFound desc = could not find container \"ce8c7fc643810362034dee469bfbab0a8ca0e12f9524cbef44879570cbbf9434\": container with ID starting with ce8c7fc643810362034dee469bfbab0a8ca0e12f9524cbef44879570cbbf9434 not found: ID does not exist"
Nov 03 20:20:17.147424 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.147201    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"]
Nov 03 20:20:17.153419 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.153394    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-c7454hndk"]
Nov 03 20:20:17.325786 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.325760    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:17.373859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.373767    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-kube-api-access-k7vc7\" (UniqueName: \"kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-kube-api-access-k7vc7\") pod \"4ef30fea-df01-4a8e-a60d-d86443df5a57\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") "
Nov 03 20:20:17.374039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.373868    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-routing-overlay-volume\") pod \"4ef30fea-df01-4a8e-a60d-d86443df5a57\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") "
Nov 03 20:20:17.374039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.374009    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-provider-credentials\") pod \"4ef30fea-df01-4a8e-a60d-d86443df5a57\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") "
Nov 03 20:20:17.374172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.374052    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-plugins-config-volume\") pod \"4ef30fea-df01-4a8e-a60d-d86443df5a57\" (UID: \"4ef30fea-df01-4a8e-a60d-d86443df5a57\") "
Nov 03 20:20:17.374437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.374397    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-routing-overlay-volume" pod "4ef30fea-df01-4a8e-a60d-d86443df5a57" (UID: "4ef30fea-df01-4a8e-a60d-d86443df5a57"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:17.374543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.374471    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-plugins-config-volume" pod "4ef30fea-df01-4a8e-a60d-d86443df5a57" (UID: "4ef30fea-df01-4a8e-a60d-d86443df5a57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:17.374996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.374968    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:17.375100 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.374998    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ef30fea-df01-4a8e-a60d-d86443df5a57-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:17.376856 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.376821    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-provider-credentials" pod "4ef30fea-df01-4a8e-a60d-d86443df5a57" (UID: "4ef30fea-df01-4a8e-a60d-d86443df5a57"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:17.384311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.384278    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-kube-api-access-k7vc7" pod "4ef30fea-df01-4a8e-a60d-d86443df5a57" (UID: "4ef30fea-df01-4a8e-a60d-d86443df5a57"). InnerVolumeSpecName "kube-api-access-k7vc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:17.477329 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.477291    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7vc7\" (UniqueName: \"kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-kube-api-access-k7vc7\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:17.477329 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.477332    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ef30fea-df01-4a8e-a60d-d86443df5a57-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:17.631399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.631302    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:17.974212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:17.974167    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e026989c-d59f-4382-b039-bebe81c37b7b" path="/var/lib/kubelet/pods/e026989c-d59f-4382-b039-bebe81c37b7b/volumes"
Nov 03 20:20:18.129033 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:18.128999    5659 generic.go:356] "Generic (PLEG): container finished" podID="4ef30fea-df01-4a8e-a60d-d86443df5a57" containerID="18c028a4f2b485ca812babad7c6869dd340ff6ff0e0f918a2e55b8eedaaf0177" exitCode=0
Nov 03 20:20:18.129247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:18.129044    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"
Nov 03 20:20:18.129247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:18.129083    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz" event={"ID":"4ef30fea-df01-4a8e-a60d-d86443df5a57","Type":"ContainerDied","Data":"18c028a4f2b485ca812babad7c6869dd340ff6ff0e0f918a2e55b8eedaaf0177"}
Nov 03 20:20:18.129247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:18.129134    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz" event={"ID":"4ef30fea-df01-4a8e-a60d-d86443df5a57","Type":"ContainerDied","Data":"5a692533abc147cbf16fb01c9023383f0a535b47d189a8096b7ce16c88b3dbb7"}
Nov 03 20:20:18.129247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:18.129162    5659 scope.go:122] "RemoveContainer" containerID="18c028a4f2b485ca812babad7c6869dd340ff6ff0e0f918a2e55b8eedaaf0177"
Nov 03 20:20:18.140673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:18.140651    5659 scope.go:122] "RemoveContainer" containerID="18c028a4f2b485ca812babad7c6869dd340ff6ff0e0f918a2e55b8eedaaf0177"
Nov 03 20:20:18.140953 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:18.140929    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c028a4f2b485ca812babad7c6869dd340ff6ff0e0f918a2e55b8eedaaf0177\": container with ID starting with 18c028a4f2b485ca812babad7c6869dd340ff6ff0e0f918a2e55b8eedaaf0177 not found: ID does not exist" containerID="18c028a4f2b485ca812babad7c6869dd340ff6ff0e0f918a2e55b8eedaaf0177"
Nov 03 20:20:18.141048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:18.140960    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c028a4f2b485ca812babad7c6869dd340ff6ff0e0f918a2e55b8eedaaf0177"} err="failed to get container status \"18c028a4f2b485ca812babad7c6869dd340ff6ff0e0f918a2e55b8eedaaf0177\": rpc error: code = NotFound desc = could not find container \"18c028a4f2b485ca812babad7c6869dd340ff6ff0e0f918a2e55b8eedaaf0177\": container with ID starting with 18c028a4f2b485ca812babad7c6869dd340ff6ff0e0f918a2e55b8eedaaf0177 not found: ID does not exist"
Nov 03 20:20:18.151651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:18.151616    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"]
Nov 03 20:20:18.157914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:18.157882    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-6f6bd2lhz"]
Nov 03 20:20:18.350547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:18.350424    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:18.491533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:18.491499    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:18.626872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:18.626793    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:19.058027 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:19.057987    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:19.794773 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:19.794362    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:19.972675 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:19.972637    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef30fea-df01-4a8e-a60d-d86443df5a57" path="/var/lib/kubelet/pods/4ef30fea-df01-4a8e-a60d-d86443df5a57/volumes"
Nov 03 20:20:20.525598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:20.525563    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:21.305246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:21.305209    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:22.038851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:22.038804    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:22.742117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:22.742085    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:23.429040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:23.428998    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:24.116292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:24.115861    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:24.864730 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:24.864686    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:25.510769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:25.510734    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:25.663998 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:25.663943    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:25.808558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:25.807466    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:26.000501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:26.000043    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:26.281028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:26.280997    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:26.446601 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:26.446563    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:26.863619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:26.863569    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:27.014330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.014287    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:27.727379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.725895    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:27.735516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.731029    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"]
Nov 03 20:20:27.741333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.741305    5659 cpu_manager.go:414] "containerMap: removing container" podUID="4ef30fea-df01-4a8e-a60d-d86443df5a57" containerName="payload-processing"
Nov 03 20:20:27.741549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.741533    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef30fea-df01-4a8e-a60d-d86443df5a57" containerName="payload-processing"
Nov 03 20:20:27.741700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.741689    5659 cpu_manager.go:414] "containerMap: removing container" podUID="e026989c-d59f-4382-b039-bebe81c37b7b" containerName="payload-processing"
Nov 03 20:20:27.741796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.741785    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e026989c-d59f-4382-b039-bebe81c37b7b" containerName="payload-processing"
Nov 03 20:20:27.742136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.742124    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="e026989c-d59f-4382-b039-bebe81c37b7b" containerName="payload-processing"
Nov 03 20:20:27.742278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.742266    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef30fea-df01-4a8e-a60d-d86443df5a57" containerName="payload-processing"
Nov 03 20:20:27.748582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.748557    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"]
Nov 03 20:20:27.748706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.748687    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:27.754408 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.754387    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"routing-overlay\""
Nov 03 20:20:27.754588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.754425    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"agc-cred-60dec61e44e9\""
Nov 03 20:20:27.755465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.755444    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"openshift-service-ca.crt\""
Nov 03 20:20:27.755649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.755571    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4c2a28\""
Nov 03 20:20:27.755759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.755653    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"kube-root-ca.crt\""
Nov 03 20:20:27.852512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.852459    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:27.937520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.931771    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v69pw\" (UniqueName: \"kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-kube-api-access-v69pw\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:27.937520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.931913    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:27.937520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.932091    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:27.937520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:27.932184    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:28.041835 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.041743    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:28.042152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.042131    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:28.042455 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.042438    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v69pw\" (UniqueName: \"kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-kube-api-access-v69pw\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:28.042713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.042694    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:28.043920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.043896    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:28.052277 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.052250    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:28.057513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.057477    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"agc-cred-60dec61e44e9\""
Nov 03 20:20:28.109812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.071290    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:28.126807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.118443    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v69pw\" (UniqueName: \"kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-kube-api-access-v69pw\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:28.280988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.280954    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"]
Nov 03 20:20:28.289977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.289949    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:28.304122 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.304028    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"agc-cred-e5af0b75dc63\""
Nov 03 20:20:28.307182 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.307152    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"openshift-service-ca.crt\""
Nov 03 20:20:28.307351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.307253    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"payload-processing-external-model-e2e--aa51d60f-dockercfg-wjs86\""
Nov 03 20:20:28.307351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.307327    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"payload-processing-external-model-plugins-e2e-extmod-a-33e267\""
Nov 03 20:20:28.307537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.307390    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"routing-overlay\""
Nov 03 20:20:28.307537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.307464    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"kube-root-ca.crt\""
Nov 03 20:20:28.308751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.308715    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:28.361504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.343437    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"]
Nov 03 20:20:28.392561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.387679    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:28.422441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.413394    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:28.422441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.413558    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:28.422441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.413644    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l8l8\" (UniqueName: \"kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-kube-api-access-5l8l8\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:28.422441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.413828    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:28.540630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.532673    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l8l8\" (UniqueName: \"kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-kube-api-access-5l8l8\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:28.540630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.532855    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:28.540630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.533076    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:28.540630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.533178    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:28.562558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.541196    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:28.611505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.608040    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:28.824562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.813707    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:28.824562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:28.814725    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:29.107552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.105903    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l8l8\" (UniqueName: \"kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-kube-api-access-5l8l8\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:29.174571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.154697    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:29.244076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.244038    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:29.393747 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.393620    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:29.450728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.450682    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:29.675572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.657439    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:29.787586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.785156    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:29.825342 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.825293    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:29.828782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.828760    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"]
Nov 03 20:20:29.850522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.848339    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:29.855510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.850782    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"]
Nov 03 20:20:29.855510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.851048    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:29.920915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.898580    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:29.920915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.915656    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h8gx\" (UniqueName: \"kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-kube-api-access-8h8gx\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:29.920915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.915859    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:29.920915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.915925    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:29.920915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.916123    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:29.959638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:29.956594    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.030931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.030900    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"payload-processing-external-model-e2e--088821c0-dockercfg-jwvtb\""
Nov 03 20:20:30.031423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.031393    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"agc-cred-bc1cba871723\""
Nov 03 20:20:30.046282 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.046245    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h8gx\" (UniqueName: \"kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-kube-api-access-8h8gx\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:30.048710 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.048678    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:30.050473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.050337    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:30.050721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.050698    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:30.050994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.050972    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:30.051990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.051970    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:30.056014 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.055994    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:30.064954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.064561    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h8gx\" (UniqueName: \"kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-kube-api-access-8h8gx\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:30.256701 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.256647    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:30.312717 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.312469    5659 scope.go:122] "RemoveContainer" containerID="8d020de094144ab7f0abf14cee2895b13fe7608593f314d1471f5d86e5d77f35"
Nov 03 20:20:30.334762 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:20:30.331127    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3f65a4_80a7_48e4_aac8_7f13fd8ef3bd.slice/crio-9e84b9266a36f0aa92be27cb81bcec13b3e3bb195a2a4f8160929fe153bd9371 WatchSource:0}: Error finding container 9e84b9266a36f0aa92be27cb81bcec13b3e3bb195a2a4f8160929fe153bd9371: Status 404 returned error can't find the container with id 9e84b9266a36f0aa92be27cb81bcec13b3e3bb195a2a4f8160929fe153bd9371
Nov 03 20:20:30.334762 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.333820    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"]
Nov 03 20:20:30.340900 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.340864    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"]
Nov 03 20:20:30.360148 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.360115    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:30.379168 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.379142    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:30.975169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:30.975070    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:31.297631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:31.297594    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf" event={"ID":"a3a64c89-9dda-4fa6-946c-962cf2c02e5f","Type":"ContainerStarted","Data":"f0560c900bc9eb3c156b64130c7fb2e31a12b8379b45ae7c7b8e7b0c91e1fb23"}
Nov 03 20:20:31.298089 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:31.297653    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf" event={"ID":"a3a64c89-9dda-4fa6-946c-962cf2c02e5f","Type":"ContainerStarted","Data":"4d7181af41cc0ee91d2190fe346afce60df72c9a1eead1d810642249a5a22504"}
Nov 03 20:20:31.298089 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:31.297680    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:31.299415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:31.299380    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc" event={"ID":"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd","Type":"ContainerStarted","Data":"931e82676e8e09f37f98e64194c71cd03db771f7fc66f39d605b4ed1699158e3"}
Nov 03 20:20:31.299551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:31.299438    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc" event={"ID":"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd","Type":"ContainerStarted","Data":"9e84b9266a36f0aa92be27cb81bcec13b3e3bb195a2a4f8160929fe153bd9371"}
Nov 03 20:20:31.299551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:31.299461    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:31.318988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:31.318933    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf" podStartSLOduration=3.318916002 podStartE2EDuration="3.318916002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:31.316602006 +0000 UTC m=+2583.471504437" watchObservedRunningTime="2026-11-03 20:20:31.318916002 +0000 UTC m=+2583.473818423"
Nov 03 20:20:31.341624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:31.341564    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc" podStartSLOduration=4.34154675 podStartE2EDuration="4.34154675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:31.338722884 +0000 UTC m=+2583.493625304" watchObservedRunningTime="2026-11-03 20:20:31.34154675 +0000 UTC m=+2583.496449169"
Nov 03 20:20:31.405396 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:20:31.405366    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc864e33e_c887_4ef9_a277_a203c7c507fd.slice/crio-c9e2563ea8db516f15ec0791541d0424fbab5900830a285d275ab6e042a96aa3 WatchSource:0}: Error finding container c9e2563ea8db516f15ec0791541d0424fbab5900830a285d275ab6e042a96aa3: Status 404 returned error can't find the container with id c9e2563ea8db516f15ec0791541d0424fbab5900830a285d275ab6e042a96aa3
Nov 03 20:20:31.405643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:31.405619    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"]
Nov 03 20:20:31.672593 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:31.672469    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:32.308827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:32.308788    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq" event={"ID":"c864e33e-c887-4ef9-a277-a203c7c507fd","Type":"ContainerStarted","Data":"705181f3a711e0e031c107af526ade6e2fb7fab08e751118503a98c66f29b1ed"}
Nov 03 20:20:32.309337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:32.308844    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq" event={"ID":"c864e33e-c887-4ef9-a277-a203c7c507fd","Type":"ContainerStarted","Data":"c9e2563ea8db516f15ec0791541d0424fbab5900830a285d275ab6e042a96aa3"}
Nov 03 20:20:32.309337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:32.308876    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:32.330021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:32.329968    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq" podStartSLOduration=3.329955464 podStartE2EDuration="3.329955464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:32.32714118 +0000 UTC m=+2584.482043600" watchObservedRunningTime="2026-11-03 20:20:32.329955464 +0000 UTC m=+2584.484857882"
Nov 03 20:20:32.435843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:32.435805    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:33.160509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:33.149387    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:33.831106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:33.831069    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:34.567816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:34.567778    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:35.248980 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:35.248947    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:35.409872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:35.409843    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:35.555567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:35.549836    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:36.042753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:36.042708    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:36.665820 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:36.665782    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:36.814218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:36.808298    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:37.452913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:37.452873    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:38.128719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:38.128687    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:38.827403 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:38.827371    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:39.089139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:39.089057    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:39.959048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:39.959014    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:41.124682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:41.124651    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:41.647672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:41.647638    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:41.790321 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:41.790280    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:42.343728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:42.343693    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:20:42.345926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:42.345898    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:43.330901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:43.330846    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:20:43.589566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:43.565512    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"]
Nov 03 20:20:44.940196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:44.940141    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:45.139398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:45.138930    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:45.288077 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:45.288043    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:45.640255 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:45.640129    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:20:45.682929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:45.682896    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:45.832504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:45.824772    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:46.336649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:46.336613    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:47.121426 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:47.121392    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:47.768670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:47.768637    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:48.474292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.474252    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:48.586276 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.586224    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc" podUID="9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd" containerName="payload-processing" containerID="cri-o://931e82676e8e09f37f98e64194c71cd03db771f7fc66f39d605b4ed1699158e3" gracePeriod=25
Nov 03 20:20:48.844973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.844939    5659 generic.go:356] "Generic (PLEG): container finished" podID="9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd" containerID="931e82676e8e09f37f98e64194c71cd03db771f7fc66f39d605b4ed1699158e3" exitCode=0
Nov 03 20:20:48.845442 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.845016    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc" event={"ID":"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd","Type":"ContainerDied","Data":"931e82676e8e09f37f98e64194c71cd03db771f7fc66f39d605b4ed1699158e3"}
Nov 03 20:20:48.916126 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.916097    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:48.958076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.958023    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-kube-api-access-v69pw\" (UniqueName: \"kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-kube-api-access-v69pw\") pod \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") "
Nov 03 20:20:48.958275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.958087    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-plugins-config-volume\") pod \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") "
Nov 03 20:20:48.958275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.958229    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-provider-credentials\") pod \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") "
Nov 03 20:20:48.958275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.958265    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-routing-overlay-volume\") pod \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\" (UID: \"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd\") "
Nov 03 20:20:48.959081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.958763    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-plugins-config-volume" pod "9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd" (UID: "9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:48.959217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.959113    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-routing-overlay-volume" pod "9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd" (UID: "9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:48.959846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.959776    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.959846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.959806    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.964462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.964437    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-provider-credentials" pod "9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd" (UID: "9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:48.964608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:48.964475    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-kube-api-access-v69pw" pod "9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd" (UID: "9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd"). InnerVolumeSpecName "kube-api-access-v69pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:49.061833 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:49.061742    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v69pw\" (UniqueName: \"kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-kube-api-access-v69pw\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:49.061833 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:49.061774    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:49.194074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:49.194042    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:49.855663 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:49.855630    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"
Nov 03 20:20:49.855663 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:49.855639    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc" event={"ID":"9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd","Type":"ContainerDied","Data":"9e84b9266a36f0aa92be27cb81bcec13b3e3bb195a2a4f8160929fe153bd9371"}
Nov 03 20:20:49.856182 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:49.855746    5659 scope.go:122] "RemoveContainer" containerID="931e82676e8e09f37f98e64194c71cd03db771f7fc66f39d605b4ed1699158e3"
Nov 03 20:20:49.883616 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:49.883585    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"]
Nov 03 20:20:49.888825 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:49.888802    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7c47q76qc"]
Nov 03 20:20:49.944143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:49.944102    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:50.026500 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:50.026423    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd" path="/var/lib/kubelet/pods/9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd/volumes"
Nov 03 20:20:50.630176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:50.630141    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:51.338807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:51.338770    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:52.088364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:52.088332    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:52.243005 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:52.242972    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:52.469105 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:52.469071    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:52.625307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:52.625274    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:52.831656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:52.831570    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:53.289140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:53.289105    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:53.435246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:53.435210    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:53.560911 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:53.560825    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:54.221848 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.221808    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:54.803068 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.803033    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"]
Nov 03 20:20:54.804372 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.804343    5659 cpu_manager.go:414] "containerMap: removing container" podUID="9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd" containerName="payload-processing"
Nov 03 20:20:54.804372 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.804372    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd" containerName="payload-processing"
Nov 03 20:20:54.804815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.804692    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3f65a4-80a7-48e4-aac8-7f13fd8ef3bd" containerName="payload-processing"
Nov 03 20:20:54.815668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.815637    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"]
Nov 03 20:20:54.815989 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.815968    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:54.829849 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.829815    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"agc-cred-567d6793aba1\""
Nov 03 20:20:54.839578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.837784    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"routing-overlay\""
Nov 03 20:20:54.868524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.840301    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"payload-processing-external-model-e2e--29763338-dockercfg-w7tmc\""
Nov 03 20:20:54.868524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.850183    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec\""
Nov 03 20:20:54.868524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.850247    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"openshift-service-ca.crt\""
Nov 03 20:20:54.868524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.850344    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"kube-root-ca.crt\""
Nov 03 20:20:54.913505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.911789    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:55.000390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.992088    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqbmz\" (UniqueName: \"kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-kube-api-access-zqbmz\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:55.000390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.992249    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:55.000390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.992409    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:55.000390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:54.992538    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:55.119641 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:55.117253    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:55.181501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:55.179024    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:55.181501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:55.179178    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqbmz\" (UniqueName: \"kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-kube-api-access-zqbmz\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:55.181501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:55.179301    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:55.181501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:55.179474    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:55.181501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:55.180415    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:55.181501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:55.181156    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:55.221277 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:55.219900    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:55.225682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:55.225619    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqbmz\" (UniqueName: \"kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-kube-api-access-zqbmz\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:55.399284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:55.399200    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:55.677224 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:55.677109    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:55.698800 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:55.698767    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"]
Nov 03 20:20:55.698800 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:20:55.698783    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37ce92a_20b8_4f5d_af7a_e4c4950e4a7e.slice/crio-bf82f54583c2152424b587ad1ed9a5485dc4d96470bab385eb9c82f5735ed212 WatchSource:0}: Error finding container bf82f54583c2152424b587ad1ed9a5485dc4d96470bab385eb9c82f5735ed212: Status 404 returned error can't find the container with id bf82f54583c2152424b587ad1ed9a5485dc4d96470bab385eb9c82f5735ed212
Nov 03 20:20:55.824711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:55.824620    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:56.109562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:56.105150    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"]
Nov 03 20:20:56.246847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:56.246809    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:56.252526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:56.252471    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" event={"ID":"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e","Type":"ContainerStarted","Data":"641b751383c4327fedad4ce732d9b70b804f765cee7dddcd35fc9e42f0ebae9e"}
Nov 03 20:20:56.252702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:56.252596    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" event={"ID":"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e","Type":"ContainerStarted","Data":"bf82f54583c2152424b587ad1ed9a5485dc4d96470bab385eb9c82f5735ed212"}
Nov 03 20:20:56.253046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:56.253023    5659 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" secret="" err="secret \"payload-processing-external-model-e2e--29763338-dockercfg-w7tmc\" not found"
Nov 03 20:20:56.253733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:56.253710    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:20:56.280511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:56.279402    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" podStartSLOduration=2.279379137 podStartE2EDuration="2.279379137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:56.274423401 +0000 UTC m=+2608.429325821" watchObservedRunningTime="2026-11-03 20:20:56.279379137 +0000 UTC m=+2608.434281558"
Nov 03 20:20:56.350958 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:56.350920    5659 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec: configmap "payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec" not found
Nov 03 20:20:56.351236 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:56.351218    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume podName:d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e nodeName:}" failed. No retries permitted until 2026-11-03 20:20:56.851194513 +0000 UTC m=+2609.006096925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" (UID: "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec" not found
Nov 03 20:20:56.351691 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:56.351668    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a0d4ec/agc-cred-567d6793aba1: secret "agc-cred-567d6793aba1" not found
Nov 03 20:20:56.358581 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:56.358554    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt: secret "agc-cred-567d6793aba1" not found
Nov 03 20:20:56.358862 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:56.358846    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials podName:d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e nodeName:}" failed. No retries permitted until 2026-11-03 20:20:56.858806994 +0000 UTC m=+2609.013709406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" (UID: "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e") : secret "agc-cred-567d6793aba1" not found
Nov 03 20:20:56.473504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:56.463580    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:56.763458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:56.763142    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"]
Nov 03 20:20:56.878380 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:56.878335    5659 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec: configmap "payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec" not found
Nov 03 20:20:56.878846 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:56.878414    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume podName:d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e nodeName:}" failed. No retries permitted until 2026-11-03 20:20:57.878393493 +0000 UTC m=+2610.033295905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" (UID: "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec" not found
Nov 03 20:20:56.878846 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:56.878493    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a0d4ec/agc-cred-567d6793aba1: secret "agc-cred-567d6793aba1" not found
Nov 03 20:20:56.878846 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:56.878515    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt: secret "agc-cred-567d6793aba1" not found
Nov 03 20:20:56.878846 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:56.878576    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials podName:d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e nodeName:}" failed. No retries permitted until 2026-11-03 20:20:57.878558481 +0000 UTC m=+2610.033460884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" (UID: "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e") : secret "agc-cred-567d6793aba1" not found
Nov 03 20:20:56.967785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:56.967574    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:57.651971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:57.651932    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:57.920477 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:57.920382    5659 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec: configmap "payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec" not found
Nov 03 20:20:57.920477 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:57.920464    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume podName:d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e nodeName:}" failed. No retries permitted until 2026-11-03 20:20:59.920443052 +0000 UTC m=+2612.075345471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" (UID: "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec" not found
Nov 03 20:20:57.920978 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:57.920517    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a0d4ec/agc-cred-567d6793aba1: secret "agc-cred-567d6793aba1" not found
Nov 03 20:20:57.920978 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:57.920532    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt: secret "agc-cred-567d6793aba1" not found
Nov 03 20:20:57.920978 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:57.920571    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials podName:d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e nodeName:}" failed. No retries permitted until 2026-11-03 20:20:59.920560003 +0000 UTC m=+2612.075462409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" (UID: "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e") : secret "agc-cred-567d6793aba1" not found
Nov 03 20:20:58.444951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:58.444905    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:58.693943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:58.693906    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"]
Nov 03 20:20:59.181503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:59.181435    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:59.829467 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:20:59.829432    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:59.970862 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:59.970806    5659 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec: configmap "payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec" not found
Nov 03 20:20:59.971088 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:59.970911    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume podName:d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e nodeName:}" failed. No retries permitted until 2026-11-03 20:21:03.97089036 +0000 UTC m=+2616.125792769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" (UID: "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec" not found
Nov 03 20:20:59.971088 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:59.970980    5659 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a0d4ec/agc-cred-567d6793aba1: secret "agc-cred-567d6793aba1" not found
Nov 03 20:20:59.971088 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:59.971002    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt: secret "agc-cred-567d6793aba1" not found
Nov 03 20:20:59.971088 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:20:59.971056    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials podName:d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e nodeName:}" failed. No retries permitted until 2026-11-03 20:21:03.971039439 +0000 UTC m=+2616.125941849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" (UID: "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e") : secret "agc-cred-567d6793aba1" not found
Nov 03 20:21:00.591999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:00.591963    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:01.475511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:01.474220    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:01.764314 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:01.764204    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf" podUID="a3a64c89-9dda-4fa6-946c-962cf2c02e5f" containerName="payload-processing" containerID="cri-o://f0560c900bc9eb3c156b64130c7fb2e31a12b8379b45ae7c7b8e7b0c91e1fb23" gracePeriod=25
Nov 03 20:21:02.233837 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.233787    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:21:02.253021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.252978    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-kube-api-access-5l8l8\" (UniqueName: \"kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-kube-api-access-5l8l8\") pod \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") "
Nov 03 20:21:02.253604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.253574    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-routing-overlay-volume\") pod \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") "
Nov 03 20:21:02.254090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.254064    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-provider-credentials\") pod \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") "
Nov 03 20:21:02.257520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.254286    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-plugins-config-volume\") pod \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\" (UID: \"a3a64c89-9dda-4fa6-946c-962cf2c02e5f\") "
Nov 03 20:21:02.272520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.258862    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-routing-overlay-volume" pod "a3a64c89-9dda-4fa6-946c-962cf2c02e5f" (UID: "a3a64c89-9dda-4fa6-946c-962cf2c02e5f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:02.272520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.259910    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-plugins-config-volume" pod "a3a64c89-9dda-4fa6-946c-962cf2c02e5f" (UID: "a3a64c89-9dda-4fa6-946c-962cf2c02e5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:02.272520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.272064    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" podUID="d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e" containerName="payload-processing" containerID="cri-o://641b751383c4327fedad4ce732d9b70b804f765cee7dddcd35fc9e42f0ebae9e" gracePeriod=25
Nov 03 20:21:02.308593 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.302209    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:02.308593 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.304656    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-provider-credentials" pod "a3a64c89-9dda-4fa6-946c-962cf2c02e5f" (UID: "a3a64c89-9dda-4fa6-946c-962cf2c02e5f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:02.340152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.333210    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-kube-api-access-5l8l8" pod "a3a64c89-9dda-4fa6-946c-962cf2c02e5f" (UID: "a3a64c89-9dda-4fa6-946c-962cf2c02e5f"). InnerVolumeSpecName "kube-api-access-5l8l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:02.358737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.358637    5659 generic.go:356] "Generic (PLEG): container finished" podID="a3a64c89-9dda-4fa6-946c-962cf2c02e5f" containerID="f0560c900bc9eb3c156b64130c7fb2e31a12b8379b45ae7c7b8e7b0c91e1fb23" exitCode=0
Nov 03 20:21:02.359338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.359063    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf" event={"ID":"a3a64c89-9dda-4fa6-946c-962cf2c02e5f","Type":"ContainerDied","Data":"f0560c900bc9eb3c156b64130c7fb2e31a12b8379b45ae7c7b8e7b0c91e1fb23"}
Nov 03 20:21:02.359338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.359133    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf" event={"ID":"a3a64c89-9dda-4fa6-946c-962cf2c02e5f","Type":"ContainerDied","Data":"4d7181af41cc0ee91d2190fe346afce60df72c9a1eead1d810642249a5a22504"}
Nov 03 20:21:02.359338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.359167    5659 scope.go:122] "RemoveContainer" containerID="f0560c900bc9eb3c156b64130c7fb2e31a12b8379b45ae7c7b8e7b0c91e1fb23"
Nov 03 20:21:02.360184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.359657    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"
Nov 03 20:21:02.518659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.503648    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:02.518659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.503684    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:02.518659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.503702    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l8l8\" (UniqueName: \"kubernetes.io/projected/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-kube-api-access-5l8l8\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:02.518659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.503718    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3a64c89-9dda-4fa6-946c-962cf2c02e5f-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:02.581398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.578862    5659 scope.go:122] "RemoveContainer" containerID="f0560c900bc9eb3c156b64130c7fb2e31a12b8379b45ae7c7b8e7b0c91e1fb23"
Nov 03 20:21:02.581398 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:02.579326    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0560c900bc9eb3c156b64130c7fb2e31a12b8379b45ae7c7b8e7b0c91e1fb23\": container with ID starting with f0560c900bc9eb3c156b64130c7fb2e31a12b8379b45ae7c7b8e7b0c91e1fb23 not found: ID does not exist" containerID="f0560c900bc9eb3c156b64130c7fb2e31a12b8379b45ae7c7b8e7b0c91e1fb23"
Nov 03 20:21:02.581398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.579363    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0560c900bc9eb3c156b64130c7fb2e31a12b8379b45ae7c7b8e7b0c91e1fb23"} err="failed to get container status \"f0560c900bc9eb3c156b64130c7fb2e31a12b8379b45ae7c7b8e7b0c91e1fb23\": rpc error: code = NotFound desc = could not find container \"f0560c900bc9eb3c156b64130c7fb2e31a12b8379b45ae7c7b8e7b0c91e1fb23\": container with ID starting with f0560c900bc9eb3c156b64130c7fb2e31a12b8379b45ae7c7b8e7b0c91e1fb23 not found: ID does not exist"
Nov 03 20:21:02.589096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.589062    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"]
Nov 03 20:21:02.594208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.594173    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-758dmf2qf"]
Nov 03 20:21:02.873251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.873220    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:21:02.922909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.922869    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-kube-api-access-zqbmz\" (UniqueName: \"kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-kube-api-access-zqbmz\") pod \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") "
Nov 03 20:21:02.923180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.923110    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume\") pod \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") "
Nov 03 20:21:02.923180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.923148    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials\") pod \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") "
Nov 03 20:21:02.923336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.923235    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-routing-overlay-volume\") pod \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\" (UID: \"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e\") "
Nov 03 20:21:02.923774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.923746    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume" pod "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e" (UID: "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:02.923888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.923811    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-routing-overlay-volume" pod "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e" (UID: "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:02.924151 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.924124    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:02.924242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.924156    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:02.932910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.932830    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-kube-api-access-zqbmz" pod "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e" (UID: "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e"). InnerVolumeSpecName "kube-api-access-zqbmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:02.933337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:02.933312    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials" pod "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e" (UID: "d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:03.026614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.026580    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqbmz\" (UniqueName: \"kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-kube-api-access-zqbmz\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:03.026614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.026613    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:03.061216 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.061154    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:03.210239 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.210178    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:03.360248 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.360213    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:03.370248 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.370207    5659 generic.go:356] "Generic (PLEG): container finished" podID="d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e" containerID="641b751383c4327fedad4ce732d9b70b804f765cee7dddcd35fc9e42f0ebae9e" exitCode=0
Nov 03 20:21:03.370456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.370258    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"
Nov 03 20:21:03.370456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.370289    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" event={"ID":"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e","Type":"ContainerDied","Data":"641b751383c4327fedad4ce732d9b70b804f765cee7dddcd35fc9e42f0ebae9e"}
Nov 03 20:21:03.370456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.370349    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt" event={"ID":"d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e","Type":"ContainerDied","Data":"bf82f54583c2152424b587ad1ed9a5485dc4d96470bab385eb9c82f5735ed212"}
Nov 03 20:21:03.370456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.370377    5659 scope.go:122] "RemoveContainer" containerID="641b751383c4327fedad4ce732d9b70b804f765cee7dddcd35fc9e42f0ebae9e"
Nov 03 20:21:03.383464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.383422    5659 scope.go:122] "RemoveContainer" containerID="641b751383c4327fedad4ce732d9b70b804f765cee7dddcd35fc9e42f0ebae9e"
Nov 03 20:21:03.383805 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:03.383779    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641b751383c4327fedad4ce732d9b70b804f765cee7dddcd35fc9e42f0ebae9e\": container with ID starting with 641b751383c4327fedad4ce732d9b70b804f765cee7dddcd35fc9e42f0ebae9e not found: ID does not exist" containerID="641b751383c4327fedad4ce732d9b70b804f765cee7dddcd35fc9e42f0ebae9e"
Nov 03 20:21:03.383890 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.383814    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641b751383c4327fedad4ce732d9b70b804f765cee7dddcd35fc9e42f0ebae9e"} err="failed to get container status \"641b751383c4327fedad4ce732d9b70b804f765cee7dddcd35fc9e42f0ebae9e\": rpc error: code = NotFound desc = could not find container \"641b751383c4327fedad4ce732d9b70b804f765cee7dddcd35fc9e42f0ebae9e\": container with ID starting with 641b751383c4327fedad4ce732d9b70b804f765cee7dddcd35fc9e42f0ebae9e not found: ID does not exist"
Nov 03 20:21:03.402010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.401978    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"]
Nov 03 20:21:03.409356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.409322    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7448gs2gt"]
Nov 03 20:21:03.550933 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.550830    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:03.695191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:03.695149    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq" podUID="c864e33e-c887-4ef9-a277-a203c7c507fd" containerName="payload-processing" containerID="cri-o://705181f3a711e0e031c107af526ade6e2fb7fab08e751118503a98c66f29b1ed" gracePeriod=25
Nov 03 20:21:04.004091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.004046    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a64c89-9dda-4fa6-946c-962cf2c02e5f" path="/var/lib/kubelet/pods/a3a64c89-9dda-4fa6-946c-962cf2c02e5f/volumes"
Nov 03 20:21:04.005497 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.005458    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e" path="/var/lib/kubelet/pods/d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e/volumes"
Nov 03 20:21:04.060184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.060115    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:04.068413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.068325    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:21:04.204542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.192137    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-kube-api-access-8h8gx\" (UniqueName: \"kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-kube-api-access-8h8gx\") pod \"c864e33e-c887-4ef9-a277-a203c7c507fd\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") "
Nov 03 20:21:04.225514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.217948    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-routing-overlay-volume" pod "c864e33e-c887-4ef9-a277-a203c7c507fd" (UID: "c864e33e-c887-4ef9-a277-a203c7c507fd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:04.468564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.438858    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-routing-overlay-volume\") pod \"c864e33e-c887-4ef9-a277-a203c7c507fd\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") "
Nov 03 20:21:04.468564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.439020    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-plugins-config-volume\") pod \"c864e33e-c887-4ef9-a277-a203c7c507fd\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") "
Nov 03 20:21:04.468564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.439330    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-provider-credentials\") pod \"c864e33e-c887-4ef9-a277-a203c7c507fd\" (UID: \"c864e33e-c887-4ef9-a277-a203c7c507fd\") "
Nov 03 20:21:04.468564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.457455    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-plugins-config-volume" pod "c864e33e-c887-4ef9-a277-a203c7c507fd" (UID: "c864e33e-c887-4ef9-a277-a203c7c507fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:04.566506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.542741    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:04.566506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.542783    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c864e33e-c887-4ef9-a277-a203c7c507fd-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:04.566506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.543264    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:04.861254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.753877    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:04.938203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.933835    5659 generic.go:356] "Generic (PLEG): container finished" podID="c864e33e-c887-4ef9-a277-a203c7c507fd" containerID="705181f3a711e0e031c107af526ade6e2fb7fab08e751118503a98c66f29b1ed" exitCode=0
Nov 03 20:21:04.938203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.933990    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq" event={"ID":"c864e33e-c887-4ef9-a277-a203c7c507fd","Type":"ContainerDied","Data":"705181f3a711e0e031c107af526ade6e2fb7fab08e751118503a98c66f29b1ed"}
Nov 03 20:21:04.938203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.934045    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq" event={"ID":"c864e33e-c887-4ef9-a277-a203c7c507fd","Type":"ContainerDied","Data":"c9e2563ea8db516f15ec0791541d0424fbab5900830a285d275ab6e042a96aa3"}
Nov 03 20:21:04.938203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.934078    5659 scope.go:122] "RemoveContainer" containerID="705181f3a711e0e031c107af526ade6e2fb7fab08e751118503a98c66f29b1ed"
Nov 03 20:21:04.938203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:04.934198    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"
Nov 03 20:21:05.013797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:05.013769    5659 scope.go:122] "RemoveContainer" containerID="705181f3a711e0e031c107af526ade6e2fb7fab08e751118503a98c66f29b1ed"
Nov 03 20:21:05.040635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:05.022499    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-kube-api-access-8h8gx" pod "c864e33e-c887-4ef9-a277-a203c7c507fd" (UID: "c864e33e-c887-4ef9-a277-a203c7c507fd"). InnerVolumeSpecName "kube-api-access-8h8gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:05.041281 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:05.041250    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705181f3a711e0e031c107af526ade6e2fb7fab08e751118503a98c66f29b1ed\": container with ID starting with 705181f3a711e0e031c107af526ade6e2fb7fab08e751118503a98c66f29b1ed not found: ID does not exist" containerID="705181f3a711e0e031c107af526ade6e2fb7fab08e751118503a98c66f29b1ed"
Nov 03 20:21:05.041461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:05.041434    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705181f3a711e0e031c107af526ade6e2fb7fab08e751118503a98c66f29b1ed"} err="failed to get container status \"705181f3a711e0e031c107af526ade6e2fb7fab08e751118503a98c66f29b1ed\": rpc error: code = NotFound desc = could not find container \"705181f3a711e0e031c107af526ade6e2fb7fab08e751118503a98c66f29b1ed\": container with ID starting with 705181f3a711e0e031c107af526ade6e2fb7fab08e751118503a98c66f29b1ed not found: ID does not exist"
Nov 03 20:21:05.050306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:05.050270    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h8gx\" (UniqueName: \"kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-kube-api-access-8h8gx\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:05.183910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:05.176288    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-provider-credentials" pod "c864e33e-c887-4ef9-a277-a203c7c507fd" (UID: "c864e33e-c887-4ef9-a277-a203c7c507fd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:05.183910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:05.183634    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c864e33e-c887-4ef9-a277-a203c7c507fd-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:05.332743 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:05.332702    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"]
Nov 03 20:21:05.337803 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:05.337765    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-7bc85grwq"]
Nov 03 20:21:06.054684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:06.054643    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c864e33e-c887-4ef9-a277-a203c7c507fd" path="/var/lib/kubelet/pods/c864e33e-c887-4ef9-a277-a203c7c507fd/volumes"
Nov 03 20:21:06.078627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:06.078537    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:06.280772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:06.280735    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:06.430228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:06.430120    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:06.714999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:06.714957    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:06.843701 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:06.843667    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:07.259913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:07.259879    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:07.399905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:07.399830    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:08.321602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.321567    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:08.464018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.463970    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:08.540086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.540050    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"]
Nov 03 20:21:08.541307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.541284    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c864e33e-c887-4ef9-a277-a203c7c507fd" containerName="payload-processing"
Nov 03 20:21:08.541307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.541306    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c864e33e-c887-4ef9-a277-a203c7c507fd" containerName="payload-processing"
Nov 03 20:21:08.541536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.541388    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e" containerName="payload-processing"
Nov 03 20:21:08.541536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.541396    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e" containerName="payload-processing"
Nov 03 20:21:08.541536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.541420    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a3a64c89-9dda-4fa6-946c-962cf2c02e5f" containerName="payload-processing"
Nov 03 20:21:08.541536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.541439    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a64c89-9dda-4fa6-946c-962cf2c02e5f" containerName="payload-processing"
Nov 03 20:21:08.541771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.541655    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37ce92a-20b8-4f5d-af7a-e4c4950e4a7e" containerName="payload-processing"
Nov 03 20:21:08.541771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.541669    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a64c89-9dda-4fa6-946c-962cf2c02e5f" containerName="payload-processing"
Nov 03 20:21:08.541771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.541687    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c864e33e-c887-4ef9-a277-a203c7c507fd" containerName="payload-processing"
Nov 03 20:21:08.547366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.547341    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:08.554847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.554813    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"]
Nov 03 20:21:08.557720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.557697    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"openshift-service-ca.crt\""
Nov 03 20:21:08.558080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.557987    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"agc-cred-220b365d4b6f\""
Nov 03 20:21:08.558498 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.558288    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"kube-root-ca.crt\""
Nov 03 20:21:08.558634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.558373    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"payload-processing-external-model-e2e--e41c9c2e-dockercfg-fvqg4\""
Nov 03 20:21:08.558739 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.558406    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-eb69cb\""
Nov 03 20:21:08.558833 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.558440    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"routing-overlay\""
Nov 03 20:21:08.586552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.585472    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:08.586552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.585635    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:08.586552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.585729    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:08.586552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.585895    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg7tk\" (UniqueName: \"kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-kube-api-access-dg7tk\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:08.611373 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.611343    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:08.719504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.715391    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:08.719504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.715644    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg7tk\" (UniqueName: \"kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-kube-api-access-dg7tk\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:08.719504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.715806    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:08.719504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.715938    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:08.719504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.716475    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:08.737515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.725275    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:08.737805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.725919    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:08.785556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.755251    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg7tk\" (UniqueName: \"kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-kube-api-access-dg7tk\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:08.914176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.914090    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:08.979917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:08.919016    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:09.156508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.129317    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"]
Nov 03 20:21:09.378645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.378609    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:09.390225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.390113    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"agc-cred-a78ddddda145\""
Nov 03 20:21:09.395509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.391790    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"payload-processing-external-model-e2e--a1bcdb2a-dockercfg-rbxx7\""
Nov 03 20:21:09.395509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.391870    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"kube-root-ca.crt\""
Nov 03 20:21:09.395509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.391956    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"routing-overlay\""
Nov 03 20:21:09.395509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.391998    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-46a6d7\""
Nov 03 20:21:09.428010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.427970    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"openshift-service-ca.crt\""
Nov 03 20:21:09.540262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.530146    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"]
Nov 03 20:21:09.574139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.573081    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:09.574139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.573226    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:09.574139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.573274    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9pnf\" (UniqueName: \"kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-kube-api-access-c9pnf\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:09.574139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.573305    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:09.704326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.704274    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:09.704520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.704417    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:09.704520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.704464    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9pnf\" (UniqueName: \"kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-kube-api-access-c9pnf\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:09.704520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.704514    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:09.711531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.705602    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:09.711531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.707804    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:09.743216 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.743181    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:09.744894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.744863    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9pnf\" (UniqueName: \"kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-kube-api-access-c9pnf\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:09.759023 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.758696    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:09.900610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.900532    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"]
Nov 03 20:21:09.901204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.901154    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84" podUID="040df513-c1c5-424f-aa74-89c927dc28be" containerName="istio-proxy" containerID="cri-o://75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d" gracePeriod=30
Nov 03 20:21:09.997656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:09.997623    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"]
Nov 03 20:21:10.029766 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.029734    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:10.096741 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.096710    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"]
Nov 03 20:21:10.097161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.097131    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:10.105010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.104988    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"
Nov 03 20:21:10.112839 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.112760    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e7b70d\""
Nov 03 20:21:10.136583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.129694    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"]
Nov 03 20:21:10.195054 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.195020    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jlv9\" (UniqueName: \"kubernetes.io/projected/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-kube-api-access-6jlv9\") pod \"payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb\" (UID: \"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"
Nov 03 20:21:10.195354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.195322    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb\" (UID: \"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"
Nov 03 20:21:10.275519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.264583    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:10.330506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.327124    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlv9\" (UniqueName: \"kubernetes.io/projected/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-kube-api-access-6jlv9\") pod \"payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb\" (UID: \"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"
Nov 03 20:21:10.330506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.327204    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb\" (UID: \"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"
Nov 03 20:21:10.330506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.328459    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb\" (UID: \"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"
Nov 03 20:21:10.502598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.502544    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf" event={"ID":"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791","Type":"ContainerStarted","Data":"ca86dc5527fb311812aa5034aaca8f709c3aea8f9903140ef4037f163a976cdd"}
Nov 03 20:21:10.503243 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.503219    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf" event={"ID":"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791","Type":"ContainerStarted","Data":"25fd4cd009dccdc3a63f4d4dee602a4d06fd9cfb7bf87fa55f7b63f1bdd22756"}
Nov 03 20:21:10.504022 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.504005    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:10.567992 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.536632    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf" podStartSLOduration=2.536611302 podStartE2EDuration="2.536611302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:10.536044455 +0000 UTC m=+2622.690946878" watchObservedRunningTime="2026-11-03 20:21:10.536611302 +0000 UTC m=+2622.691513722"
Nov 03 20:21:10.567992 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.553138    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jlv9\" (UniqueName: \"kubernetes.io/projected/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-kube-api-access-6jlv9\") pod \"payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb\" (UID: \"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"
Nov 03 20:21:10.635466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.635428    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"]
Nov 03 20:21:10.725553 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:21:10.715386    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ed1541_9d28_454b_913c_210fd8f409ac.slice/crio-340d92344a17917c43f85568957de2322adcc7a5444a670105e76ec7045840df WatchSource:0}: Error finding container 340d92344a17917c43f85568957de2322adcc7a5444a670105e76ec7045840df: Status 404 returned error can't find the container with id 340d92344a17917c43f85568957de2322adcc7a5444a670105e76ec7045840df
Nov 03 20:21:10.748616 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.747337    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"
Nov 03 20:21:10.884445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:10.884408    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:11.059282 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:21:11.059238    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e6d798_5d9f_4ee9_94c5_7d7faf5caf59.slice/crio-9a75b946dd7ae4d3be827f4f30cc9e362e7d32c9929a1cdb353e4d5cc4edb163 WatchSource:0}: Error finding container 9a75b946dd7ae4d3be827f4f30cc9e362e7d32c9929a1cdb353e4d5cc4edb163: Status 404 returned error can't find the container with id 9a75b946dd7ae4d3be827f4f30cc9e362e7d32c9929a1cdb353e4d5cc4edb163
Nov 03 20:21:11.060043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:11.060016    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"]
Nov 03 20:21:11.139074 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:11.139054    5659 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ee9d78131bed: secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:11.139074 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:11.139073    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6: secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:11.139284 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:11.139134    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials podName:67208b39-1303-4730-abc7-5e86f2547dc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:21:11.639112687 +0000 UTC m=+2623.794015103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials") pod "payload-processing-external-model-588d8bd94c-4cxs6" (UID: "67208b39-1303-4730-abc7-5e86f2547dc8") : secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:11.511765 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:11.511724    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9" event={"ID":"19ed1541-9d28-454b-913c-210fd8f409ac","Type":"ContainerStarted","Data":"8ef826c0dbbbcab17fcc55fa18d15c61a5470a2f6bd1c00198bc0ffa8fe15df6"}
Nov 03 20:21:11.512237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:11.511831    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9" event={"ID":"19ed1541-9d28-454b-913c-210fd8f409ac","Type":"ContainerStarted","Data":"340d92344a17917c43f85568957de2322adcc7a5444a670105e76ec7045840df"}
Nov 03 20:21:11.512237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:11.511994    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:11.513465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:11.513436    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb" event={"ID":"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59","Type":"ContainerStarted","Data":"08e52b666c849af3e926daa4fe9db4a2a063c21af606eea0aa9c3e451967d0a3"}
Nov 03 20:21:11.513598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:11.513508    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb" event={"ID":"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59","Type":"ContainerStarted","Data":"9a75b946dd7ae4d3be827f4f30cc9e362e7d32c9929a1cdb353e4d5cc4edb163"}
Nov 03 20:21:11.513598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:11.513537    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"
Nov 03 20:21:11.532495 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:11.532432    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9" podStartSLOduration=3.532415521 podStartE2EDuration="3.532415521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:11.531053402 +0000 UTC m=+2623.685955826" watchObservedRunningTime="2026-11-03 20:21:11.532415521 +0000 UTC m=+2623.687317942"
Nov 03 20:21:11.552348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:11.552292    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb" podStartSLOduration=1.5522769570000001 podStartE2EDuration="1.552276957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:11.551510917 +0000 UTC m=+2623.706413338" watchObservedRunningTime="2026-11-03 20:21:11.552276957 +0000 UTC m=+2623.707179377"
Nov 03 20:21:11.564710 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:11.564676    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:11.650248 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:11.650212    5659 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ee9d78131bed: secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:11.650248 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:11.650242    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6: secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:11.650471 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:11.650314    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials podName:67208b39-1303-4730-abc7-5e86f2547dc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:21:12.650293009 +0000 UTC m=+2624.805195420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials") pod "payload-processing-external-model-588d8bd94c-4cxs6" (UID: "67208b39-1303-4730-abc7-5e86f2547dc8") : secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:12.261993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:12.261955    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:12.670689 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:12.670603    5659 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ee9d78131bed: secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:12.670689 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:12.670630    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6: secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:12.671122 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:12.670693    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials podName:67208b39-1303-4730-abc7-5e86f2547dc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:21:14.670678552 +0000 UTC m=+2626.825580962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials") pod "payload-processing-external-model-588d8bd94c-4cxs6" (UID: "67208b39-1303-4730-abc7-5e86f2547dc8") : secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:12.999462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:12.999428    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:13.719340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:13.719300    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:14.362044 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:14.361993    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:14.719941 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:14.719905    5659 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ee9d78131bed: secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:14.719941 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:14.719931    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6: secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:14.720461 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:14.719987    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials podName:67208b39-1303-4730-abc7-5e86f2547dc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:21:18.719972741 +0000 UTC m=+2630.874875138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials") pod "payload-processing-external-model-588d8bd94c-4cxs6" (UID: "67208b39-1303-4730-abc7-5e86f2547dc8") : secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:14.989996 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:14.989924    5659 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040df513_c1c5_424f_aa74_89c927dc28be.slice/crio-conmon-75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:21:15.100041 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.100003    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:15.243923 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:15.243837    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:21:15.322315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.322284    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:21:15.476315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.476279    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-data\") pod \"040df513-c1c5-424f-aa74-89c927dc28be\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") "
Nov 03 20:21:15.476611 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.476586    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-data" pod "040df513-c1c5-424f-aa74-89c927dc28be" (UID: "040df513-c1c5-424f-aa74-89c927dc28be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:15.476708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.476599    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-socket\") pod \"040df513-c1c5-424f-aa74-89c927dc28be\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") "
Nov 03 20:21:15.476824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.476807    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-envoy\") pod \"040df513-c1c5-424f-aa74-89c927dc28be\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") "
Nov 03 20:21:15.476908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.476866    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-socket" pod "040df513-c1c5-424f-aa74-89c927dc28be" (UID: "040df513-c1c5-424f-aa74-89c927dc28be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:15.476908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.476877    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-istio-token\" (UniqueName: \"kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-istio-token\") pod \"040df513-c1c5-424f-aa74-89c927dc28be\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") "
Nov 03 20:21:15.477042 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.477022    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-kube-api-access-v8mxx\" (UniqueName: \"kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-kube-api-access-v8mxx\") pod \"040df513-c1c5-424f-aa74-89c927dc28be\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") "
Nov 03 20:21:15.477307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.477284    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-credential-socket\") pod \"040df513-c1c5-424f-aa74-89c927dc28be\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") "
Nov 03 20:21:15.477394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.477320    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/040df513-c1c5-424f-aa74-89c927dc28be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040df513-c1c5-424f-aa74-89c927dc28be-istio-podinfo\") pod \"040df513-c1c5-424f-aa74-89c927dc28be\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") "
Nov 03 20:21:15.477465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.477426    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-certs\") pod \"040df513-c1c5-424f-aa74-89c927dc28be\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") "
Nov 03 20:21:15.477584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.477507    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040df513-c1c5-424f-aa74-89c927dc28be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040df513-c1c5-424f-aa74-89c927dc28be-istiod-ca-cert\") pod \"040df513-c1c5-424f-aa74-89c927dc28be\" (UID: \"040df513-c1c5-424f-aa74-89c927dc28be\") "
Nov 03 20:21:15.478079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.477992    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-certs" pod "040df513-c1c5-424f-aa74-89c927dc28be" (UID: "040df513-c1c5-424f-aa74-89c927dc28be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:15.478079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.478014    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040df513-c1c5-424f-aa74-89c927dc28be-istiod-ca-cert" pod "040df513-c1c5-424f-aa74-89c927dc28be" (UID: "040df513-c1c5-424f-aa74-89c927dc28be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:15.478079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.478049    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-credential-socket" pod "040df513-c1c5-424f-aa74-89c927dc28be" (UID: "040df513-c1c5-424f-aa74-89c927dc28be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:15.478399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.478379    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:15.478531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.478403    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:15.478531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.478417    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:15.478531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.478432    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040df513-c1c5-424f-aa74-89c927dc28be-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:15.478531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.478445    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:15.479908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.479850    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-envoy" pod "040df513-c1c5-424f-aa74-89c927dc28be" (UID: "040df513-c1c5-424f-aa74-89c927dc28be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:15.480243 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.480215    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-istio-token" pod "040df513-c1c5-424f-aa74-89c927dc28be" (UID: "040df513-c1c5-424f-aa74-89c927dc28be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:15.480798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.480765    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-kube-api-access-v8mxx" pod "040df513-c1c5-424f-aa74-89c927dc28be" (UID: "040df513-c1c5-424f-aa74-89c927dc28be"). InnerVolumeSpecName "kube-api-access-v8mxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:15.480884 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.480865    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/040df513-c1c5-424f-aa74-89c927dc28be-istio-podinfo" pod "040df513-c1c5-424f-aa74-89c927dc28be" (UID: "040df513-c1c5-424f-aa74-89c927dc28be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:21:15.551648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.551557    5659 generic.go:356] "Generic (PLEG): container finished" podID="040df513-c1c5-424f-aa74-89c927dc28be" containerID="75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d" exitCode=0
Nov 03 20:21:15.551648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.551602    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"
Nov 03 20:21:15.551894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.551645    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84" event={"ID":"040df513-c1c5-424f-aa74-89c927dc28be","Type":"ContainerDied","Data":"75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d"}
Nov 03 20:21:15.551894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.551711    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84" event={"ID":"040df513-c1c5-424f-aa74-89c927dc28be","Type":"ContainerDied","Data":"fcc4b799cbb55c71ce38cf67e1834b5e2b2ac31bae8b602ccab16e1ca5bfbbc0"}
Nov 03 20:21:15.551894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.551740    5659 scope.go:122] "RemoveContainer" containerID="75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d"
Nov 03 20:21:15.564451 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.564431    5659 scope.go:122] "RemoveContainer" containerID="75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d"
Nov 03 20:21:15.564808 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:15.564785    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d\": container with ID starting with 75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d not found: ID does not exist" containerID="75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d"
Nov 03 20:21:15.564888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.564820    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d"} err="failed to get container status \"75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d\": rpc error: code = NotFound desc = could not find container \"75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d\": container with ID starting with 75f28db78a1ebedbe94bcc0ec31d17a73dbfca7e4d096da020b86cbf0333250d not found: ID does not exist"
Nov 03 20:21:15.579069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.579037    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"]
Nov 03 20:21:15.580353 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.580324    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8mxx\" (UniqueName: \"kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-kube-api-access-v8mxx\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:15.580449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.580360    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040df513-c1c5-424f-aa74-89c927dc28be-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:15.580449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.580379    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040df513-c1c5-424f-aa74-89c927dc28be-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:15.580449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.580392    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040df513-c1c5-424f-aa74-89c927dc28be-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:15.583589 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.583568    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-86g84"]
Nov 03 20:21:15.930594 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:15.930461    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:16.012733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:16.012677    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040df513-c1c5-424f-aa74-89c927dc28be" path="/var/lib/kubelet/pods/040df513-c1c5-424f-aa74-89c927dc28be/volumes"
Nov 03 20:21:16.656506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:16.653564    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:17.360651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:17.360614    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:18.048209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:18.048173    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:18.604416 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:18.604378    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"]
Nov 03 20:21:18.737603 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:18.737559    5659 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ee9d78131bed: secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:18.737603 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:18.737589    5659 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6: secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:18.737970 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:18.737644    5659 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials podName:67208b39-1303-4730-abc7-5e86f2547dc8 nodeName:}" failed. No retries permitted until 2026-11-03 20:21:26.737628694 +0000 UTC m=+2638.892531104 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials") pod "payload-processing-external-model-588d8bd94c-4cxs6" (UID: "67208b39-1303-4730-abc7-5e86f2547dc8") : secret "agc-cred-ee9d78131bed" not found
Nov 03 20:21:18.873496 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:18.865916    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:19.297341 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.297300    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"]
Nov 03 20:21:19.298136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.298118    5659 cpu_manager.go:414] "containerMap: removing container" podUID="040df513-c1c5-424f-aa74-89c927dc28be" containerName="istio-proxy"
Nov 03 20:21:19.298197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.298137    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040df513-c1c5-424f-aa74-89c927dc28be" containerName="istio-proxy"
Nov 03 20:21:19.298302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.298293    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="040df513-c1c5-424f-aa74-89c927dc28be" containerName="istio-proxy"
Nov 03 20:21:19.303860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.303838    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.312565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.312538    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4c2a28-openshift-default-dockercfg-vhjbn\""
Nov 03 20:21:19.316126 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.316097    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"]
Nov 03 20:21:19.353897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.353854    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-token\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.354096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.353915    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5sj\" (UniqueName: \"kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-kube-api-access-cb5sj\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.354096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.354071    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-envoy\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.354234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.354116    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-certs\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.354234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.354228    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.354350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.354315    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-podinfo\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.354436 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.354419    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-data\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.354542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.354518    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-credential-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.354602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.354552    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba237d4-70b3-4347-86b6-462fc504fe3d-istiod-ca-cert\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.456500 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.456450    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5sj\" (UniqueName: \"kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-kube-api-access-cb5sj\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.456730 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.456574    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-envoy\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.456730 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.456626    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-certs\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.456874 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.456761    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.456933 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.456873    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-podinfo\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.457018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.457001    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-data\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.457095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.457078    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-credential-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.457159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.457116    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba237d4-70b3-4347-86b6-462fc504fe3d-istiod-ca-cert\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.457227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.457201    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.458109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.457392    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-token\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.458109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.457428    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-certs\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.458109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.457469    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-credential-socket\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.458109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.457776    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-data\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.458424 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.458121    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba237d4-70b3-4347-86b6-462fc504fe3d-istiod-ca-cert\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.459369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.459345    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-envoy\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.460516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.460475    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-podinfo\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.469698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.469670    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-token\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.469822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.469762    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5sj\" (UniqueName: \"kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-kube-api-access-cb5sj\") pod \"e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") " pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.521834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.521792    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:19.621719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.621327    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:19.951380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.951344    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"]
Nov 03 20:21:19.955622 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.955580    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:19.955758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.955670    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:19.955758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:19.955736    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:20.267040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:20.266991    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:20.630445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:20.630345    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c" event={"ID":"aba237d4-70b3-4347-86b6-462fc504fe3d","Type":"ContainerStarted","Data":"697a722756df43548d3fc79ede2b5977cae0a2e6906a2a790ac5a1c955d4bf77"}
Nov 03 20:21:20.630445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:20.630417    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c" event={"ID":"aba237d4-70b3-4347-86b6-462fc504fe3d","Type":"ContainerStarted","Data":"a6a9ea2791e953e49792b0e7431c803498ba7783c985d481dde3175f9de1cba9"}
Nov 03 20:21:20.662544 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:20.662466    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c" podStartSLOduration=1.662447148 podStartE2EDuration="1.662447148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:20.654406409 +0000 UTC m=+2632.809308830" watchObservedRunningTime="2026-11-03 20:21:20.662447148 +0000 UTC m=+2632.817349569"
Nov 03 20:21:20.967561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:20.951998    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:21.516268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:21.516237    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:21:21.622338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:21.622302    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:21.623831 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:21.623805    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:21:21.623958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:21.623862    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c" podUID="aba237d4-70b3-4347-86b6-462fc504fe3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 20:21:21.685504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:21.683036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:22.330687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:22.330653    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:22.524816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:22.524786    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"
Nov 03 20:21:22.524995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:22.524860    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:21:22.622865 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:22.622775    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:21:22.623108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:22.623082    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c" podUID="aba237d4-70b3-4347-86b6-462fc504fe3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 20:21:23.064770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:23.064719    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:23.605521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:23.605466    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6" podUID="67208b39-1303-4730-abc7-5e86f2547dc8" containerName="payload-processing" containerID="cri-o://c1a49fdb950b76c8a580e765dbd424cff93f693aac607d6d2ce6bd39ec35c3b5" gracePeriod=25
Nov 03 20:21:23.622085 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:23.622056    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:21:23.622202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:23.622112    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c" podUID="aba237d4-70b3-4347-86b6-462fc504fe3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 20:21:23.791411 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:23.791372    5659 generic.go:356] "Generic (PLEG): container finished" podID="67208b39-1303-4730-abc7-5e86f2547dc8" containerID="c1a49fdb950b76c8a580e765dbd424cff93f693aac607d6d2ce6bd39ec35c3b5" exitCode=0
Nov 03 20:21:23.791733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:23.791442    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6" event={"ID":"67208b39-1303-4730-abc7-5e86f2547dc8","Type":"ContainerDied","Data":"c1a49fdb950b76c8a580e765dbd424cff93f693aac607d6d2ce6bd39ec35c3b5"}
Nov 03 20:21:23.831103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:23.831040    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:24.079827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.079793    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:21:24.189753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.189711    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-plugins-config-volume\") pod \"67208b39-1303-4730-abc7-5e86f2547dc8\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") "
Nov 03 20:21:24.189937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.189795    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-routing-overlay-volume\") pod \"67208b39-1303-4730-abc7-5e86f2547dc8\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") "
Nov 03 20:21:24.190018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.189997    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-kube-api-access-5hscq\" (UniqueName: \"kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-kube-api-access-5hscq\") pod \"67208b39-1303-4730-abc7-5e86f2547dc8\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") "
Nov 03 20:21:24.190138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.190108    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials\") pod \"67208b39-1303-4730-abc7-5e86f2547dc8\" (UID: \"67208b39-1303-4730-abc7-5e86f2547dc8\") "
Nov 03 20:21:24.190324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.190188    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-routing-overlay-volume" pod "67208b39-1303-4730-abc7-5e86f2547dc8" (UID: "67208b39-1303-4730-abc7-5e86f2547dc8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:24.190577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.190227    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-plugins-config-volume" pod "67208b39-1303-4730-abc7-5e86f2547dc8" (UID: "67208b39-1303-4730-abc7-5e86f2547dc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:24.191243 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.191218    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:24.191412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.191395    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67208b39-1303-4730-abc7-5e86f2547dc8-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:24.192539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.192514    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials" pod "67208b39-1303-4730-abc7-5e86f2547dc8" (UID: "67208b39-1303-4730-abc7-5e86f2547dc8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:24.192624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.192557    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-kube-api-access-5hscq" pod "67208b39-1303-4730-abc7-5e86f2547dc8" (UID: "67208b39-1303-4730-abc7-5e86f2547dc8"). InnerVolumeSpecName "kube-api-access-5hscq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:24.293916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.293822    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hscq\" (UniqueName: \"kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-kube-api-access-5hscq\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:24.293916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.293862    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67208b39-1303-4730-abc7-5e86f2547dc8-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:24.551559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.548361    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:24.630505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.624131    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:21:24.630505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.624209    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c" podUID="aba237d4-70b3-4347-86b6-462fc504fe3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 20:21:24.801385 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.801357    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"
Nov 03 20:21:24.801655 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.801383    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6" event={"ID":"67208b39-1303-4730-abc7-5e86f2547dc8","Type":"ContainerDied","Data":"bdc19f736dda12cef5186cb5d81f11c979d27418d28eda036f77b5ef6d44920b"}
Nov 03 20:21:24.801655 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.801449    5659 scope.go:122] "RemoveContainer" containerID="c1a49fdb950b76c8a580e765dbd424cff93f693aac607d6d2ce6bd39ec35c3b5"
Nov 03 20:21:24.869217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.869157    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"]
Nov 03 20:21:24.873517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:24.873469    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-588d8bd94c-4cxs6"]
Nov 03 20:21:25.356506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:25.356308    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:25.621894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:25.621823    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:21:25.621894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:25.621881    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c" podUID="aba237d4-70b3-4347-86b6-462fc504fe3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 20:21:25.975549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:25.975440    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67208b39-1303-4730-abc7-5e86f2547dc8" path="/var/lib/kubelet/pods/67208b39-1303-4730-abc7-5e86f2547dc8/volumes"
Nov 03 20:21:26.024069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:26.024034    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:26.622181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:26.622140    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:21:26.622733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:26.622220    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c" podUID="aba237d4-70b3-4347-86b6-462fc504fe3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 03 20:21:26.859656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:26.859623    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:27.837935 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:27.837902    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:27.909356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:27.909325    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:27.923289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:27.923260    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:21:28.154646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:28.151708    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:29.058822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:29.058619    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:29.233540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:29.233510    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:30.743173 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:30.743091    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:30.978641 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:30.978598    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnfsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:30.979108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:30.979070    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" podUID="9fa9e0c2-1bfa-4b41-b033-8d286b4c63be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:32.156544 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:32.156467    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:34.873568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:34.873523    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:35.343744 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:35.343709    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:36.033238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:36.033206    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:36.808011 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:36.807978    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:37.523708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:37.523681    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:38.216095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:38.216059    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:38.938883 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:38.938825    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:39.213354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:39.213327    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:39.728917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:39.728886    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:40.542460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:40.542410    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:41.222435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:41.222398    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:42.131795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:42.131759    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:42.929678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:42.929641    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:43.597184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:43.597140    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:43.766339 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:43.766302    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"]
Nov 03 20:21:43.766567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:43.766526    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj" podUID="1a4750b2-2d03-49c3-ab74-07df22747954" containerName="istio-proxy" containerID="cri-o://0e982b0fb27c6895de2690425cb489542eb08f2ce6ae2e8d0ebb5739b1cd6201" gracePeriod=30
Nov 03 20:21:44.431474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:44.431443    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:45.138786 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:45.138730    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:45.304454 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:45.304408    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:21:45.789877 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:45.789844    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:46.482155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:46.482107    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:47.235271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:47.233065    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:47.904676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:47.904632    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:48.730785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:48.730751    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:49.071618 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.071589    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:21:49.218305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.217443    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-envoy\") pod \"1a4750b2-2d03-49c3-ab74-07df22747954\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") "
Nov 03 20:21:49.218671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.218649    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-data\") pod \"1a4750b2-2d03-49c3-ab74-07df22747954\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") "
Nov 03 20:21:49.219827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.219032    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-data" pod "1a4750b2-2d03-49c3-ab74-07df22747954" (UID: "1a4750b2-2d03-49c3-ab74-07df22747954"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:49.220018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.220002    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-istio-token\" (UniqueName: \"kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-istio-token\") pod \"1a4750b2-2d03-49c3-ab74-07df22747954\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") "
Nov 03 20:21:49.222723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.221371    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4750b2-2d03-49c3-ab74-07df22747954-istiod-ca-cert" pod "1a4750b2-2d03-49c3-ab74-07df22747954" (UID: "1a4750b2-2d03-49c3-ab74-07df22747954"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:49.222723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.221778    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a4750b2-2d03-49c3-ab74-07df22747954-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4750b2-2d03-49c3-ab74-07df22747954-istiod-ca-cert\") pod \"1a4750b2-2d03-49c3-ab74-07df22747954\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") "
Nov 03 20:21:49.222723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.221879    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-credential-socket\") pod \"1a4750b2-2d03-49c3-ab74-07df22747954\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") "
Nov 03 20:21:49.222723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.221937    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-certs\") pod \"1a4750b2-2d03-49c3-ab74-07df22747954\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") "
Nov 03 20:21:49.222723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.221972    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-kube-api-access-22zkb\" (UniqueName: \"kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-kube-api-access-22zkb\") pod \"1a4750b2-2d03-49c3-ab74-07df22747954\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") "
Nov 03 20:21:49.222723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.222086    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-socket\") pod \"1a4750b2-2d03-49c3-ab74-07df22747954\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") "
Nov 03 20:21:49.222723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.222145    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a4750b2-2d03-49c3-ab74-07df22747954-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4750b2-2d03-49c3-ab74-07df22747954-istio-podinfo\") pod \"1a4750b2-2d03-49c3-ab74-07df22747954\" (UID: \"1a4750b2-2d03-49c3-ab74-07df22747954\") "
Nov 03 20:21:49.222723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.221772    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-envoy" pod "1a4750b2-2d03-49c3-ab74-07df22747954" (UID: "1a4750b2-2d03-49c3-ab74-07df22747954"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:49.222723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.222675    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-socket" pod "1a4750b2-2d03-49c3-ab74-07df22747954" (UID: "1a4750b2-2d03-49c3-ab74-07df22747954"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:49.223226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.223205    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:49.223325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.223230    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:49.223325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.223246    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4750b2-2d03-49c3-ab74-07df22747954-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:49.223325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.223261    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:49.223544 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.223327    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-certs" pod "1a4750b2-2d03-49c3-ab74-07df22747954" (UID: "1a4750b2-2d03-49c3-ab74-07df22747954"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:49.223983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.223825    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-credential-socket" pod "1a4750b2-2d03-49c3-ab74-07df22747954" (UID: "1a4750b2-2d03-49c3-ab74-07df22747954"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:21:49.223983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.223948    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-istio-token" pod "1a4750b2-2d03-49c3-ab74-07df22747954" (UID: "1a4750b2-2d03-49c3-ab74-07df22747954"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:49.225212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.225166    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-kube-api-access-22zkb" pod "1a4750b2-2d03-49c3-ab74-07df22747954" (UID: "1a4750b2-2d03-49c3-ab74-07df22747954"). InnerVolumeSpecName "kube-api-access-22zkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:49.225734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.225711    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a4750b2-2d03-49c3-ab74-07df22747954-istio-podinfo" pod "1a4750b2-2d03-49c3-ab74-07df22747954" (UID: "1a4750b2-2d03-49c3-ab74-07df22747954"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:21:49.325415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.325331    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4750b2-2d03-49c3-ab74-07df22747954-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:49.325415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.325362    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:49.325415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.325375    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:49.325415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.325383    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4750b2-2d03-49c3-ab74-07df22747954-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:49.325415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.325396    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22zkb\" (UniqueName: \"kubernetes.io/projected/1a4750b2-2d03-49c3-ab74-07df22747954-kube-api-access-22zkb\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:49.344613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.344574    5659 generic.go:356] "Generic (PLEG): container finished" podID="1a4750b2-2d03-49c3-ab74-07df22747954" containerID="0e982b0fb27c6895de2690425cb489542eb08f2ce6ae2e8d0ebb5739b1cd6201" exitCode=0
Nov 03 20:21:49.344795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.344628    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"
Nov 03 20:21:49.344795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.344661    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj" event={"ID":"1a4750b2-2d03-49c3-ab74-07df22747954","Type":"ContainerDied","Data":"0e982b0fb27c6895de2690425cb489542eb08f2ce6ae2e8d0ebb5739b1cd6201"}
Nov 03 20:21:49.344795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.344722    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj" event={"ID":"1a4750b2-2d03-49c3-ab74-07df22747954","Type":"ContainerDied","Data":"3f9e10f49f15280c5bac537f517e1dd636b529eca2b41699910b11e413ed8b16"}
Nov 03 20:21:49.344795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.344751    5659 scope.go:122] "RemoveContainer" containerID="0e982b0fb27c6895de2690425cb489542eb08f2ce6ae2e8d0ebb5739b1cd6201"
Nov 03 20:21:49.357870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.357849    5659 scope.go:122] "RemoveContainer" containerID="0e982b0fb27c6895de2690425cb489542eb08f2ce6ae2e8d0ebb5739b1cd6201"
Nov 03 20:21:49.358172 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:21:49.358153    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e982b0fb27c6895de2690425cb489542eb08f2ce6ae2e8d0ebb5739b1cd6201\": container with ID starting with 0e982b0fb27c6895de2690425cb489542eb08f2ce6ae2e8d0ebb5739b1cd6201 not found: ID does not exist" containerID="0e982b0fb27c6895de2690425cb489542eb08f2ce6ae2e8d0ebb5739b1cd6201"
Nov 03 20:21:49.358227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.358183    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e982b0fb27c6895de2690425cb489542eb08f2ce6ae2e8d0ebb5739b1cd6201"} err="failed to get container status \"0e982b0fb27c6895de2690425cb489542eb08f2ce6ae2e8d0ebb5739b1cd6201\": rpc error: code = NotFound desc = could not find container \"0e982b0fb27c6895de2690425cb489542eb08f2ce6ae2e8d0ebb5739b1cd6201\": container with ID starting with 0e982b0fb27c6895de2690425cb489542eb08f2ce6ae2e8d0ebb5739b1cd6201 not found: ID does not exist"
Nov 03 20:21:49.375334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.375270    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"]
Nov 03 20:21:49.385391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.385350    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-ltmdj"]
Nov 03 20:21:49.444853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.444816    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:49.971174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:49.971128    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4750b2-2d03-49c3-ab74-07df22747954" path="/var/lib/kubelet/pods/1a4750b2-2d03-49c3-ab74-07df22747954/volumes"
Nov 03 20:21:50.205965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:50.205930    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:50.897581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:50.894897    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:51.872801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:51.872768    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:52.584581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:52.584547    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:53.326667 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:53.326635    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:53.837366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:53.837327    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:54.043992 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.043960    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:54.313264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.313231    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:54.688698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.688601    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"]
Nov 03 20:21:54.689842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.689815    5659 cpu_manager.go:414] "containerMap: removing container" podUID="67208b39-1303-4730-abc7-5e86f2547dc8" containerName="payload-processing"
Nov 03 20:21:54.689962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.689845    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67208b39-1303-4730-abc7-5e86f2547dc8" containerName="payload-processing"
Nov 03 20:21:54.689962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.689867    5659 cpu_manager.go:414] "containerMap: removing container" podUID="1a4750b2-2d03-49c3-ab74-07df22747954" containerName="istio-proxy"
Nov 03 20:21:54.689962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.689875    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4750b2-2d03-49c3-ab74-07df22747954" containerName="istio-proxy"
Nov 03 20:21:54.690124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.690117    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="67208b39-1303-4730-abc7-5e86f2547dc8" containerName="payload-processing"
Nov 03 20:21:54.690179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.690131    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4750b2-2d03-49c3-ab74-07df22747954" containerName="istio-proxy"
Nov 03 20:21:54.695226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.695202    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.712059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.712025    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-74dfc6-openshift-default-dockercfg-btjcs\""
Nov 03 20:21:54.725716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.725681    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"]
Nov 03 20:21:54.753095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.753057    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:54.755938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.755844    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-socket\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.756127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.756111    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-credential-socket\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.756249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.756238    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-data\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.756341 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.756330    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bslwm\" (UniqueName: \"kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-kube-api-access-bslwm\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.756425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.756416    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdffad9b-17e0-408b-b096-73ca58b439b3-istiod-ca-cert\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.756534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.756524    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-certs\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.756628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.756617    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-envoy\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.756830 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.756819    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-podinfo\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.756987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.756977    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-token\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.909873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.909833    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-podinfo\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.927733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.927648    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-token\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.930525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.930475    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-socket\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.931197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.931175    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-credential-socket\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.940862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.940801    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-data\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.941360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.941341    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bslwm\" (UniqueName: \"kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-kube-api-access-bslwm\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.945670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.945644    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdffad9b-17e0-408b-b096-73ca58b439b3-istiod-ca-cert\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.977811 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.977773    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-certs\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:54.993740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.993706    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-envoy\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:55.007058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.931105    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-socket\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:55.041798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.946513    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdffad9b-17e0-408b-b096-73ca58b439b3-istiod-ca-cert\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:55.066226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.959912    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:55.066597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.941260    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-data\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:55.066739 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.978403    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-certs\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:55.066945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:54.931629    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-credential-socket\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:55.067102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:55.000414    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bslwm\" (UniqueName: \"kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-kube-api-access-bslwm\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:55.067196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:55.066102    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-token\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:55.094503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:55.094304    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-podinfo\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:55.121508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:55.115327    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-envoy\") pod \"e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") " pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:55.261995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:55.261938    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:55.348538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:55.346759    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:55.653154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:55.653056    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:55.700378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:55.699709    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"]
Nov 03 20:21:55.751662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:55.750915    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:55.751662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:55.751046    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:55.751662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:55.751091    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:56.439682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:56.439643    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:56.511953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:56.509351    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" event={"ID":"fdffad9b-17e0-408b-b096-73ca58b439b3","Type":"ContainerStarted","Data":"30e83865b7ebf30fc3a0e9227f984d63c3fb78162ab1d055f998b6b84f917ead"}
Nov 03 20:21:56.511953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:56.509422    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" event={"ID":"fdffad9b-17e0-408b-b096-73ca58b439b3","Type":"ContainerStarted","Data":"0fd896136965acafe1ae1f76000a5636a3799b1d0dbe3f1b0f423f429f53b3c5"}
Nov 03 20:21:56.547742 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:56.547674    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" podStartSLOduration=2.547651567 podStartE2EDuration="2.547651567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:56.543820655 +0000 UTC m=+2668.698723075" watchObservedRunningTime="2026-11-03 20:21:56.547651567 +0000 UTC m=+2668.702553987"
Nov 03 20:21:57.253505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:57.250204    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:57.351920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:57.351886    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:57.362804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:57.362770    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:57.518441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:57.518366    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:57.519526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:57.519469    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:21:58.365517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:58.358385    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:59.026856 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:59.026821    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:59.266508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:21:59.265840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:00.028879 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.028843    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:00.280751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.280653    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:00.296356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.296322    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"]
Nov 03 20:22:00.301713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.301688    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"
Nov 03 20:22:00.308737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.308709    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-74dfc6\""
Nov 03 20:22:00.311083 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.311055    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"]
Nov 03 20:22:00.345374 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.345331    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"]
Nov 03 20:22:00.351351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.351310    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"
Nov 03 20:22:00.371124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.371093    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-74dfc6-dockercfg-9ddm6\""
Nov 03 20:22:00.381537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.379821    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"]
Nov 03 20:22:00.463506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.463022    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5dac14-76e3-44c6-9926-44b0156716c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl\" (UID: \"4d5dac14-76e3-44c6-9926-44b0156716c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"
Nov 03 20:22:00.463506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.463197    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdr6t\" (UniqueName: \"kubernetes.io/projected/4d5dac14-76e3-44c6-9926-44b0156716c7-kube-api-access-bdr6t\") pod \"payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl\" (UID: \"4d5dac14-76e3-44c6-9926-44b0156716c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"
Nov 03 20:22:00.463506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.463347    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6db9932-e1b0-4a21-9874-4c63d3b40a10-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47\" (UID: \"f6db9932-e1b0-4a21-9874-4c63d3b40a10\") " pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"
Nov 03 20:22:00.463849 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.463532    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdtwd\" (UniqueName: \"kubernetes.io/projected/f6db9932-e1b0-4a21-9874-4c63d3b40a10-kube-api-access-gdtwd\") pod \"payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47\" (UID: \"f6db9932-e1b0-4a21-9874-4c63d3b40a10\") " pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"
Nov 03 20:22:00.486513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.464144    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:00.701559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.701104    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6db9932-e1b0-4a21-9874-4c63d3b40a10-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47\" (UID: \"f6db9932-e1b0-4a21-9874-4c63d3b40a10\") " pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"
Nov 03 20:22:00.701559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.701363    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdtwd\" (UniqueName: \"kubernetes.io/projected/f6db9932-e1b0-4a21-9874-4c63d3b40a10-kube-api-access-gdtwd\") pod \"payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47\" (UID: \"f6db9932-e1b0-4a21-9874-4c63d3b40a10\") " pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"
Nov 03 20:22:00.719502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.718305    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6db9932-e1b0-4a21-9874-4c63d3b40a10-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47\" (UID: \"f6db9932-e1b0-4a21-9874-4c63d3b40a10\") " pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"
Nov 03 20:22:00.719502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.719384    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5dac14-76e3-44c6-9926-44b0156716c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl\" (UID: \"4d5dac14-76e3-44c6-9926-44b0156716c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"
Nov 03 20:22:00.719780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.719572    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdr6t\" (UniqueName: \"kubernetes.io/projected/4d5dac14-76e3-44c6-9926-44b0156716c7-kube-api-access-bdr6t\") pod \"payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl\" (UID: \"4d5dac14-76e3-44c6-9926-44b0156716c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"
Nov 03 20:22:00.728505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.720832    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5dac14-76e3-44c6-9926-44b0156716c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl\" (UID: \"4d5dac14-76e3-44c6-9926-44b0156716c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"
Nov 03 20:22:00.792507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.790622    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdr6t\" (UniqueName: \"kubernetes.io/projected/4d5dac14-76e3-44c6-9926-44b0156716c7-kube-api-access-bdr6t\") pod \"payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl\" (UID: \"4d5dac14-76e3-44c6-9926-44b0156716c7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"
Nov 03 20:22:00.802542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.796268    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdtwd\" (UniqueName: \"kubernetes.io/projected/f6db9932-e1b0-4a21-9874-4c63d3b40a10-kube-api-access-gdtwd\") pod \"payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47\" (UID: \"f6db9932-e1b0-4a21-9874-4c63d3b40a10\") " pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"
Nov 03 20:22:00.923927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:00.923886    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"
Nov 03 20:22:01.001000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:01.000817    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:01.024551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:01.015804    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"
Nov 03 20:22:01.421907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:01.421879    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"]
Nov 03 20:22:01.422258 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:22:01.421904    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5dac14_76e3_44c6_9926_44b0156716c7.slice/crio-b700f3a4e41357f42eb26139ec34a191dc0f3347b11ffa9f36570759fda7e028 WatchSource:0}: Error finding container b700f3a4e41357f42eb26139ec34a191dc0f3347b11ffa9f36570759fda7e028: Status 404 returned error can't find the container with id b700f3a4e41357f42eb26139ec34a191dc0f3347b11ffa9f36570759fda7e028
Nov 03 20:22:01.432135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:01.432108    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"]
Nov 03 20:22:01.437645 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:22:01.437613    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6db9932_e1b0_4a21_9874_4c63d3b40a10.slice/crio-cd43bab1bfe92cb1f74df34fccddb6f43b603423af29b3e29af5b6ba94e520d9 WatchSource:0}: Error finding container cd43bab1bfe92cb1f74df34fccddb6f43b603423af29b3e29af5b6ba94e520d9: Status 404 returned error can't find the container with id cd43bab1bfe92cb1f74df34fccddb6f43b603423af29b3e29af5b6ba94e520d9
Nov 03 20:22:01.681418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:01.681315    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl" event={"ID":"4d5dac14-76e3-44c6-9926-44b0156716c7","Type":"ContainerStarted","Data":"5b8fd1903abbe54fde7503fb95a9b5d3030ce6ae3a762b3e65857918473c43c4"}
Nov 03 20:22:01.681418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:01.681375    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl" event={"ID":"4d5dac14-76e3-44c6-9926-44b0156716c7","Type":"ContainerStarted","Data":"b700f3a4e41357f42eb26139ec34a191dc0f3347b11ffa9f36570759fda7e028"}
Nov 03 20:22:01.681703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:01.681626    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"
Nov 03 20:22:01.683286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:01.683260    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47" event={"ID":"f6db9932-e1b0-4a21-9874-4c63d3b40a10","Type":"ContainerStarted","Data":"8d943ed7e272ada8fa824eab8dd5b58e0bc74bfc38a2454995e07b779eb5736e"}
Nov 03 20:22:01.683403 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:01.683304    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47" event={"ID":"f6db9932-e1b0-4a21-9874-4c63d3b40a10","Type":"ContainerStarted","Data":"cd43bab1bfe92cb1f74df34fccddb6f43b603423af29b3e29af5b6ba94e520d9"}
Nov 03 20:22:01.683403 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:01.683353    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"
Nov 03 20:22:01.693711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:01.693690    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:01.766101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:01.766036    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl" podStartSLOduration=1.766016231 podStartE2EDuration="1.766016231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:01.699695631 +0000 UTC m=+2673.854598049" watchObservedRunningTime="2026-11-03 20:22:01.766016231 +0000 UTC m=+2673.920918651"
Nov 03 20:22:01.828016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:01.825782    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47" podStartSLOduration=1.825763255 podStartE2EDuration="1.825763255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:01.746927239 +0000 UTC m=+2673.901829659" watchObservedRunningTime="2026-11-03 20:22:01.825763255 +0000 UTC m=+2673.980665675"
Nov 03 20:22:02.462533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:02.462468    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:03.149862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:03.149824    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:03.828578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:03.828539    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:04.469661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:04.469625    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:05.147216 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:05.147180    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:05.922381 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:05.922333    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:06.653754 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:06.653717    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:07.542232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:07.542200    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:08.380352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:08.380318    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:09.198877 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:09.198838    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:09.921691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:09.921658    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:10.734804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:10.734671    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:11.592940 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:11.592908    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:12.243854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:12.243825    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:12.695315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:12.695236    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"
Nov 03 20:22:12.695769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:12.695354    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"
Nov 03 20:22:12.943349 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:12.943312    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:13.669633 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:13.669600    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:14.537736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:14.537702    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:15.219653 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:22:15.219613    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:22:15.236130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:15.236098    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:16.031843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:16.031804    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:16.747907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:16.747873    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:17.429145 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:17.429109    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:18.054992 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:18.054957    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:18.921712 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:18.921674    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:19.069161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:19.069125    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:19.743721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:19.743672    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:20.565498 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:20.565454    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:21.357177 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:21.357136    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:22.022213 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:22.022175    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:22.780960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:22.780921    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:23.447711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:23.447672    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:24.087153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:24.087083    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:24.809956 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:24.809921    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:25.465026 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:25.464990    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:26.134063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:26.134028    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:26.850112 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:26.850074    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:27.535146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:27.535112    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:28.402022 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:28.390522    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:29.392217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:29.392181    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:30.306012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:30.305980    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:31.723742 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:31.723709    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:33.029361 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:33.029321    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:33.723903 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:33.723872    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:34.458057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:34.458022    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:34.608181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:34.608141    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:35.080518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:35.079859    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:36.062291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:36.062259    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:36.728939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:36.728900    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:37.138612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:37.138532    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:37.297889 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:37.297850    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:37.798361 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:37.798313    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:38.749254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:38.749218    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:39.168437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:39.168354    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:39.298906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:39.298871    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:39.436356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:39.436273    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:40.165989 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:40.165955    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:40.859163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:40.859052    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:41.722160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:41.722121    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:42.449914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:42.449867    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:43.494670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:43.494631    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:44.220817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:44.220783    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:44.723198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:44.723168    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:44.955272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:44.955227    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:45.219599 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:22:45.219566    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:22:45.774966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:45.774863    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:45.904505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:45.904455    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:46.629743 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:46.629709    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:47.538637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:47.538602    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:48.417538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:48.417503    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:49.148590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:49.148556    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:49.850953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:49.850918    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:50.747396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:50.747306    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:50.880354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:50.879979    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:51.033596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:51.027902    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:51.838826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:51.838785    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:52.517474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:52.517440    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:52.674686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:52.674645    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:52.818919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:52.818831    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:53.114512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:53.114415    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:53.502880 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:53.502789    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:53.712634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:53.712604    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:53.847976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:53.847875    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:54.378862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:54.378829    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:55.018509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:54.921556    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:55.194506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:55.190247    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:55.661870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:55.661820    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:56.445925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:56.445889    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:56.986623 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:56.986588    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:57.294467 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:57.294375    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:58.027630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:58.027587    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:59.062240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:22:59.062199    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:00.303509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:00.280740    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:01.144807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:01.144720    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:01.825636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:01.825598    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:02.627874 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:02.627834    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:03.449391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:03.449358    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:04.130266 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:04.130229    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:04.898071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:04.898042    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:05.662988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:05.662945    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:06.452815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:06.452781    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:07.135773 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:07.135723    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:07.838962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:07.838916    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:08.638715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:08.638677    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:09.264101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:09.264063    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:09.816384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:09.816337    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"]
Nov 03 20:23:09.890118 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:09.889916    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:10.058513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:10.035559    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:10.591541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:10.589575    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:10.762584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:10.751698    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:11.335641 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:11.335604    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:12.033723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:12.033637    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:12.175382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:12.175291    5659 generic.go:356] "Generic (PLEG): container finished" podID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerID="30e83865b7ebf30fc3a0e9227f984d63c3fb78162ab1d055f998b6b84f917ead" exitCode=0
Nov 03 20:23:12.175581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:12.175376    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" event={"ID":"fdffad9b-17e0-408b-b096-73ca58b439b3","Type":"ContainerDied","Data":"30e83865b7ebf30fc3a0e9227f984d63c3fb78162ab1d055f998b6b84f917ead"}
Nov 03 20:23:12.176032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:12.176013    5659 scope.go:122] "RemoveContainer" containerID="30e83865b7ebf30fc3a0e9227f984d63c3fb78162ab1d055f998b6b84f917ead"
Nov 03 20:23:12.178369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:12.178338    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:23:12.178466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:12.178414    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:23:12.178466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:12.178447    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:23:12.518939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:12.518903    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:23:12.928624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:12.928471    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:13.186017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:13.185927    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" event={"ID":"fdffad9b-17e0-408b-b096-73ca58b439b3","Type":"ContainerStarted","Data":"5bfd63224718fa4e97f9fd1819aaa34ddfb5a5342005c3c5e695e94a05ace7bc"}
Nov 03 20:23:13.351507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:13.349632    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:23:13.351507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:13.349758    5659 patch_prober.go:29] interesting pod/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:23:13.351507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:13.349801    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 03 20:23:13.719953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:13.719905    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:14.350506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:14.349895    5659 patch_prober.go:29] interesting pod/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:23:14.350506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:14.349966    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 03 20:23:14.491920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:14.491888    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:14.817308 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:14.817259    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg" podUID="e4500a62-d02c-4231-9774-44c4edcf44e6" containerName="payload-processing" containerID="cri-o://eb3ba78dc37dbe7c4424acde14adddd88dcc7c2256c40d77b4eb2957a2854e25" gracePeriod=25
Nov 03 20:23:15.030837 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:23:15.030807    5659 logging.go:55] [core] [Channel #4500 SubChannel #4501]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.177:50052", ServerName: "10.133.0.177:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.177:50052: connect: connection refused"
Nov 03 20:23:15.163029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.162749    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:15.375577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.361552    5659 patch_prober.go:29] interesting pod/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:23:15.375577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.361621    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 03 20:23:15.409769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.408613    5659 generic.go:356] "Generic (PLEG): container finished" podID="e4500a62-d02c-4231-9774-44c4edcf44e6" containerID="eb3ba78dc37dbe7c4424acde14adddd88dcc7c2256c40d77b4eb2957a2854e25" exitCode=0
Nov 03 20:23:15.410025 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.409997    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg" event={"ID":"e4500a62-d02c-4231-9774-44c4edcf44e6","Type":"ContainerDied","Data":"eb3ba78dc37dbe7c4424acde14adddd88dcc7c2256c40d77b4eb2957a2854e25"}
Nov 03 20:23:15.430783 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:23:15.430748    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:23:15.481968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.481939    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:23:15.525135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.525099    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-routing-overlay-volume\") pod \"e4500a62-d02c-4231-9774-44c4edcf44e6\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") "
Nov 03 20:23:15.525333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.525162    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-provider-credentials\") pod \"e4500a62-d02c-4231-9774-44c4edcf44e6\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") "
Nov 03 20:23:15.525333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.525249    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-kube-api-access-26n72\" (UniqueName: \"kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-kube-api-access-26n72\") pod \"e4500a62-d02c-4231-9774-44c4edcf44e6\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") "
Nov 03 20:23:15.525333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.525320    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-plugins-config-volume\") pod \"e4500a62-d02c-4231-9774-44c4edcf44e6\" (UID: \"e4500a62-d02c-4231-9774-44c4edcf44e6\") "
Nov 03 20:23:15.526615 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.526591    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-routing-overlay-volume" pod "e4500a62-d02c-4231-9774-44c4edcf44e6" (UID: "e4500a62-d02c-4231-9774-44c4edcf44e6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:15.527543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.527520    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-plugins-config-volume" pod "e4500a62-d02c-4231-9774-44c4edcf44e6" (UID: "e4500a62-d02c-4231-9774-44c4edcf44e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:15.528892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.528869    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-provider-credentials" pod "e4500a62-d02c-4231-9774-44c4edcf44e6" (UID: "e4500a62-d02c-4231-9774-44c4edcf44e6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:15.529648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.529625    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-kube-api-access-26n72" pod "e4500a62-d02c-4231-9774-44c4edcf44e6" (UID: "e4500a62-d02c-4231-9774-44c4edcf44e6"). InnerVolumeSpecName "kube-api-access-26n72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:15.627949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.627859    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:15.627949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.627901    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e4500a62-d02c-4231-9774-44c4edcf44e6-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:15.627949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.627916    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:15.627949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:15.627930    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26n72\" (UniqueName: \"kubernetes.io/projected/e4500a62-d02c-4231-9774-44c4edcf44e6-kube-api-access-26n72\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:16.030381 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:16.030334    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg" podUID="e4500a62-d02c-4231-9774-44c4edcf44e6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.177:50052\" within 1s: context deadline exceeded"
Nov 03 20:23:16.055018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:16.054897    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:16.076379 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:23:16.076339    5659 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4500a62_d02c_4231_9774_44c4edcf44e6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4500a62_d02c_4231_9774_44c4edcf44e6.slice/crio-a9e71c9aa6264f41b60289d982b2c7bf1103a66b9643f60240beb4921eb37491\": RecentStats: unable to find data in memory cache]"
Nov 03 20:23:16.356167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:16.356071    5659 patch_prober.go:29] interesting pod/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:23:16.364222 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:16.364185    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 03 20:23:16.418617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:16.418567    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg" event={"ID":"e4500a62-d02c-4231-9774-44c4edcf44e6","Type":"ContainerDied","Data":"a9e71c9aa6264f41b60289d982b2c7bf1103a66b9643f60240beb4921eb37491"}
Nov 03 20:23:16.418617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:16.418603    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"
Nov 03 20:23:16.419175 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:16.418641    5659 scope.go:122] "RemoveContainer" containerID="eb3ba78dc37dbe7c4424acde14adddd88dcc7c2256c40d77b4eb2957a2854e25"
Nov 03 20:23:16.472309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:16.472269    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"]
Nov 03 20:23:16.477539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:16.477509    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-9bf8jnssg"]
Nov 03 20:23:16.862508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:16.861937    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:17.022242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:17.022206    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:17.164105 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:17.163888    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:17.350509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:17.348875    5659 patch_prober.go:29] interesting pod/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:23:17.350509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:17.348938    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 03 20:23:17.553502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:17.544737    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:17.696513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:17.696018    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:17.982772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:17.982736    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:17.989464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:17.989417    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4500a62-d02c-4231-9774-44c4edcf44e6" path="/var/lib/kubelet/pods/e4500a62-d02c-4231-9774-44c4edcf44e6/volumes"
Nov 03 20:23:18.126390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:18.126357    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:18.357564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:18.354017    5659 patch_prober.go:29] interesting pod/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:23:18.357564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:18.354114    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 03 20:23:18.478174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:18.478142    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:19.160549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:19.160471    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:19.352887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:19.352857    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:23:19.353384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:19.353363    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:23:19.354144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:19.354123    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:23:19.938841 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:19.938803    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:20.717508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:20.714088    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:21.408683 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:21.408629    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:22.052341 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:22.052307    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:22.802817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:22.802787    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:23.525556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:23.525518    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:24.228660 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:24.228629    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:25.032782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:25.032748    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:25.723544 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:25.723512    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:26.658538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:26.655187    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:27.350079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:27.350040    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:28.034793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:28.034759    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:28.809105 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:28.809070    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:29.634152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:29.634082    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:30.320940 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:30.320906    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:31.135120 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:31.135078    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:31.839642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:31.839609    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:32.515307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:32.515273    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:33.247834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:33.247799    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:33.925122 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:33.925094    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:34.654888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:34.654857    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:35.357181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:35.357088    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:36.043394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:36.043359    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:36.864625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:36.864586    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:37.625228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:37.625198    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:38.431129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:38.431097    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:39.394630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:39.394055    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:39.935989 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:39.935954    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:40.364418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:40.364334    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:41.901768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:41.901735    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:43.012974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:43.012942    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:43.830708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:43.830677    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:44.551451 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:44.551410    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:44.846572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:44.846475    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:45.220135 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:23:45.220098    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:23:45.697562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:45.691012    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:46.416871 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:46.416837    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:47.091950 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:47.091902    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:47.989158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:47.989122    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:48.781624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:48.763627    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:49.456997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:49.456960    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:50.337638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:50.337564    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:51.044133 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:51.044093    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:51.721129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:51.721091    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:52.375058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:52.375023    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:53.125461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:53.125428    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:53.839082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:53.839039    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:54.514276 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:54.514244    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:55.339412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:55.339362    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:56.183117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:56.183076    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:56.879061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:56.879023    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:57.653704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:57.653669    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:58.463439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:23:58.463400    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:00.252527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:00.252492    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:00.395912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:00.395875    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:00.668823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:00.668745    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:01.457433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:01.457397    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:02.447242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:02.447201    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:03.178069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:03.177967    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:03.927814 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:03.927778    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:04.639577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:04.639540    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:05.310059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:05.310011    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:06.073793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:06.073524    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:06.744526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:06.744493    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:07.566332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:07.566293    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:08.347723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:08.346851    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:09.014907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:09.014826    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:09.844519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:09.844462    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:10.537119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:10.537084    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:11.328950 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:11.328909    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:12.050608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:12.050566    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:12.825109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:12.825074    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:13.501736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:13.501703    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:14.284566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:14.284414    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:15.077860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:15.077822    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:15.227687 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:24:15.227646    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:24:15.566331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:15.566277    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:15.972881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:15.972796    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:16.670208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:16.670168    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:17.330936 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:17.330899    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:18.047901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:18.047853    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:18.711964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:18.711924    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:19.530751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:19.530722    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:20.244805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:20.244748    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:20.949240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:20.949196    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:21.748512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:21.738561    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:22.540106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:22.540067    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:23.188497 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:23.188448    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:23.937757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:23.937720    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:24.836056 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:24.836014    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:25.640310 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:25.640281    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:26.384873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:26.384837    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:27.056504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:27.052006    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:27.746780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:27.746744    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:28.459500 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:28.459448    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:29.101108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:29.101075    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:29.870657 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:29.870618    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:30.637476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:30.637432    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:31.415453 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:31.415398    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:32.159337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:32.159296    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:32.941265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:32.941231    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:33.634931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:33.634863    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:34.353827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:34.353781    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:35.107023 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:35.106991    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:35.829129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:35.829099    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:36.563794 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:36.563744    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:37.327051 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:37.327019    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:38.136657 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:38.136618    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:38.852347 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:38.852303    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:39.557296 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:39.557254    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:40.028050 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:40.028013    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:40.192523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:40.192472    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:40.324888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:40.324796    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:40.955396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:40.955356    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:41.704728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:41.704693    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:42.871776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:42.871740    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:43.758950 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:43.758911    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:44.485799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:44.485750    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:45.218967 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:24:45.218931    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:24:45.378719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:45.378624    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:46.051995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:46.051951    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:46.955021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:46.935381    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:47.730200 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:47.730163    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:48.482742 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:48.482708    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:49.141983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:49.141918    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:49.833600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:49.833550    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:50.659931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:50.659896    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:51.427873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:51.427842    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:52.101867 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:52.101839    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:53.004505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:53.000490    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:53.643585 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:53.643549    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:54.326827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:54.326767    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:55.049055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:55.047738    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:55.728887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:55.728836    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:56.362771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:56.362737    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:57.147271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:57.147230    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:57.865197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:57.865137    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:58.123697 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.123618    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"]
Nov 03 20:24:58.124473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.124458    5659 cpu_manager.go:414] "containerMap: removing container" podUID="e4500a62-d02c-4231-9774-44c4edcf44e6" containerName="payload-processing"
Nov 03 20:24:58.124593 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.124475    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4500a62-d02c-4231-9774-44c4edcf44e6" containerName="payload-processing"
Nov 03 20:24:58.124651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.124639    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4500a62-d02c-4231-9774-44c4edcf44e6" containerName="payload-processing"
Nov 03 20:24:58.128339 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.128318    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:58.133649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.133631    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"agc-cred-e5af0b75dc63\""
Nov 03 20:24:58.134601 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.134579    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"agc-cred-53243bacc083\""
Nov 03 20:24:58.134717 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.134695    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"payload-processing-external-model-plugins-e2e-extmod-a-33e267\""
Nov 03 20:24:58.134845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.134776    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"kube-root-ca.crt\""
Nov 03 20:24:58.134845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.134823    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"openshift-service-ca.crt\""
Nov 03 20:24:58.135284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.134932    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"routing-overlay\""
Nov 03 20:24:58.136211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.136170    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"payload-processing-external-model-e2e--aa51d60f-dockercfg-wjs86\""
Nov 03 20:24:58.147936 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.144680    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:58.156924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.156894    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"]
Nov 03 20:24:58.336508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.336030    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:58.351576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.337384    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:58.351576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.337551    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:58.351576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.337646    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:58.351576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.337865    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wcfq\" (UniqueName: \"kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-kube-api-access-5wcfq\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:58.449967 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.441451    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:58.450563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.450532    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:58.450881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.446913    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:58.454296 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.450929    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wcfq\" (UniqueName: \"kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-kube-api-access-5wcfq\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:58.454753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.454728    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:58.455826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.455803    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:58.473970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.473933    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wcfq\" (UniqueName: \"kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-kube-api-access-5wcfq\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:58.485390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.485344    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:58.829507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:58.800699    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:24:59.348207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:59.348105    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:59.482793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:24:59.482761    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"]
Nov 03 20:24:59.483764 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:24:59.483729    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0531f8_3ef6_4c1a_a9cb_d14dd66058a6.slice/crio-f05ddeec452b0dfc2e9569bf8d51bb4beba90a3f47b025edb6b371f840566604 WatchSource:0}: Error finding container f05ddeec452b0dfc2e9569bf8d51bb4beba90a3f47b025edb6b371f840566604: Status 404 returned error can't find the container with id f05ddeec452b0dfc2e9569bf8d51bb4beba90a3f47b025edb6b371f840566604
Nov 03 20:25:00.073171 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:00.073125    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n" event={"ID":"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6","Type":"ContainerStarted","Data":"ea5bc50e35aee4f60757f330ac39aa187c055e6fe48c2fc9de553976d36af76e"}
Nov 03 20:25:00.073363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:00.073205    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n" event={"ID":"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6","Type":"ContainerStarted","Data":"f05ddeec452b0dfc2e9569bf8d51bb4beba90a3f47b025edb6b371f840566604"}
Nov 03 20:25:00.073363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:00.073237    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:25:00.093656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:00.093597    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n" podStartSLOduration=2.093582663 podStartE2EDuration="2.093582663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:00.091822967 +0000 UTC m=+2852.246725401" watchObservedRunningTime="2026-11-03 20:25:00.093582663 +0000 UTC m=+2852.248485121"
Nov 03 20:25:00.138147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:00.138108    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:00.860900 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:00.860858    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:01.605507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:01.602325    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:02.309412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:02.309377    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:03.100582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:03.100547    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:03.855142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:03.855108    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:04.652330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:04.652297    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:05.344828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:05.344770    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:06.040995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:06.040921    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:06.756315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:06.756279    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:07.429954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:07.429920    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:08.137644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:08.137608    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:08.592436 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:08.592400    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"]
Nov 03 20:25:08.641091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:08.641044    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:08.825829 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:08.825789    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:09.097508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:09.079649    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:09.297889 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:09.297795    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:09.433795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:09.431896    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:09.478758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:09.478548    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"]
Nov 03 20:25:09.581089 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:09.581036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:10.165407 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:10.165376    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:10.308507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:10.297220    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"]
Nov 03 20:25:10.310765 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:10.310734    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:10.347771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:10.347425    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:25:10.504561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:10.504461    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:11.435629 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:11.430013    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:11.631768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:11.631729    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:11.765764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:11.765735    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:12.242507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:12.240718    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:12.502579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:12.484039    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:12.757559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:12.752036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:13.462135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:13.459911    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:13.593768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:13.593728    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf" podUID="48f76d2c-0bf7-4ce4-9813-7edf8a1e1791" containerName="payload-processing" containerID="cri-o://ca86dc5527fb311812aa5034aaca8f709c3aea8f9903140ef4037f163a976cdd" gracePeriod=25
Nov 03 20:25:13.784714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:13.784673    5659 generic.go:356] "Generic (PLEG): container finished" podID="48f76d2c-0bf7-4ce4-9813-7edf8a1e1791" containerID="ca86dc5527fb311812aa5034aaca8f709c3aea8f9903140ef4037f163a976cdd" exitCode=0
Nov 03 20:25:13.788417 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:13.784935    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf" event={"ID":"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791","Type":"ContainerDied","Data":"ca86dc5527fb311812aa5034aaca8f709c3aea8f9903140ef4037f163a976cdd"}
Nov 03 20:25:13.900127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:13.900100    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:25:14.027804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.027701    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-routing-overlay-volume\") pod \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") "
Nov 03 20:25:14.027804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.027806    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-provider-credentials\" (UniqueName: \"kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-provider-credentials\") pod \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") "
Nov 03 20:25:14.028072 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.027932    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-plugins-config-volume\") pod \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") "
Nov 03 20:25:14.028072 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.027970    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-kube-api-access-dg7tk\" (UniqueName: \"kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-kube-api-access-dg7tk\") pod \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\" (UID: \"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791\") "
Nov 03 20:25:14.028252 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.028085    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-routing-overlay-volume" pod "48f76d2c-0bf7-4ce4-9813-7edf8a1e1791" (UID: "48f76d2c-0bf7-4ce4-9813-7edf8a1e1791"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:14.028329 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.028292    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-plugins-config-volume" pod "48f76d2c-0bf7-4ce4-9813-7edf8a1e1791" (UID: "48f76d2c-0bf7-4ce4-9813-7edf8a1e1791"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:14.028820 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.028790    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:14.028820 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.028822    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:14.030222 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.030199    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-provider-credentials" pod "48f76d2c-0bf7-4ce4-9813-7edf8a1e1791" (UID: "48f76d2c-0bf7-4ce4-9813-7edf8a1e1791"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:14.030321 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.030234    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-kube-api-access-dg7tk" pod "48f76d2c-0bf7-4ce4-9813-7edf8a1e1791" (UID: "48f76d2c-0bf7-4ce4-9813-7edf8a1e1791"). InnerVolumeSpecName "kube-api-access-dg7tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:14.130579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.130534    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:14.130579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.130568    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg7tk\" (UniqueName: \"kubernetes.io/projected/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791-kube-api-access-dg7tk\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:14.134517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.134449    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:14.479875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.479808    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9" podUID="19ed1541-9d28-454b-913c-210fd8f409ac" containerName="payload-processing" containerID="cri-o://8ef826c0dbbbcab17fcc55fa18d15c61a5470a2f6bd1c00198bc0ffa8fe15df6" gracePeriod=25
Nov 03 20:25:14.783216 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.783186    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:25:14.796640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.796615    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"
Nov 03 20:25:14.796795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.796651    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf" event={"ID":"48f76d2c-0bf7-4ce4-9813-7edf8a1e1791","Type":"ContainerDied","Data":"25fd4cd009dccdc3a63f4d4dee602a4d06fd9cfb7bf87fa55f7b63f1bdd22756"}
Nov 03 20:25:14.796795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.796708    5659 scope.go:122] "RemoveContainer" containerID="ca86dc5527fb311812aa5034aaca8f709c3aea8f9903140ef4037f163a976cdd"
Nov 03 20:25:14.798227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.798204    5659 generic.go:356] "Generic (PLEG): container finished" podID="19ed1541-9d28-454b-913c-210fd8f409ac" containerID="8ef826c0dbbbcab17fcc55fa18d15c61a5470a2f6bd1c00198bc0ffa8fe15df6" exitCode=0
Nov 03 20:25:14.798336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.798274    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"
Nov 03 20:25:14.798336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.798293    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9" event={"ID":"19ed1541-9d28-454b-913c-210fd8f409ac","Type":"ContainerDied","Data":"8ef826c0dbbbcab17fcc55fa18d15c61a5470a2f6bd1c00198bc0ffa8fe15df6"}
Nov 03 20:25:14.798446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.798345    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9" event={"ID":"19ed1541-9d28-454b-913c-210fd8f409ac","Type":"ContainerDied","Data":"340d92344a17917c43f85568957de2322adcc7a5444a670105e76ec7045840df"}
Nov 03 20:25:14.810281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.810257    5659 scope.go:122] "RemoveContainer" containerID="8ef826c0dbbbcab17fcc55fa18d15c61a5470a2f6bd1c00198bc0ffa8fe15df6"
Nov 03 20:25:14.819017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.818989    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"]
Nov 03 20:25:14.820102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.820082    5659 cpu_manager.go:414] "containerMap: removing container" podUID="48f76d2c-0bf7-4ce4-9813-7edf8a1e1791" containerName="payload-processing"
Nov 03 20:25:14.820102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.820101    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f76d2c-0bf7-4ce4-9813-7edf8a1e1791" containerName="payload-processing"
Nov 03 20:25:14.820259 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.820160    5659 cpu_manager.go:414] "containerMap: removing container" podUID="19ed1541-9d28-454b-913c-210fd8f409ac" containerName="payload-processing"
Nov 03 20:25:14.820259 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.820166    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ed1541-9d28-454b-913c-210fd8f409ac" containerName="payload-processing"
Nov 03 20:25:14.820376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.820284    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ed1541-9d28-454b-913c-210fd8f409ac" containerName="payload-processing"
Nov 03 20:25:14.820376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.820297    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f76d2c-0bf7-4ce4-9813-7edf8a1e1791" containerName="payload-processing"
Nov 03 20:25:14.825819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.825802    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:14.831209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.828608    5659 scope.go:122] "RemoveContainer" containerID="8ef826c0dbbbcab17fcc55fa18d15c61a5470a2f6bd1c00198bc0ffa8fe15df6"
Nov 03 20:25:14.831209 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:25:14.829164    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef826c0dbbbcab17fcc55fa18d15c61a5470a2f6bd1c00198bc0ffa8fe15df6\": container with ID starting with 8ef826c0dbbbcab17fcc55fa18d15c61a5470a2f6bd1c00198bc0ffa8fe15df6 not found: ID does not exist" containerID="8ef826c0dbbbcab17fcc55fa18d15c61a5470a2f6bd1c00198bc0ffa8fe15df6"
Nov 03 20:25:14.831209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.829201    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef826c0dbbbcab17fcc55fa18d15c61a5470a2f6bd1c00198bc0ffa8fe15df6"} err="failed to get container status \"8ef826c0dbbbcab17fcc55fa18d15c61a5470a2f6bd1c00198bc0ffa8fe15df6\": rpc error: code = NotFound desc = could not find container \"8ef826c0dbbbcab17fcc55fa18d15c61a5470a2f6bd1c00198bc0ffa8fe15df6\": container with ID starting with 8ef826c0dbbbcab17fcc55fa18d15c61a5470a2f6bd1c00198bc0ffa8fe15df6 not found: ID does not exist"
Nov 03 20:25:14.831706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.831657    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e7b70d-openshift-default-dockercfg-jjqkc\""
Nov 03 20:25:14.844368 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.843567    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"]
Nov 03 20:25:14.850475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.850321    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"]
Nov 03 20:25:14.851299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.851275    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-plugins-config-volume\") pod \"19ed1541-9d28-454b-913c-210fd8f409ac\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") "
Nov 03 20:25:14.851521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.851465    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-provider-credentials\") pod \"19ed1541-9d28-454b-913c-210fd8f409ac\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") "
Nov 03 20:25:14.851600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.851568    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-kube-api-access-c9pnf\" (UniqueName: \"kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-kube-api-access-c9pnf\") pod \"19ed1541-9d28-454b-913c-210fd8f409ac\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") "
Nov 03 20:25:14.851744 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.851716    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-routing-overlay-volume\") pod \"19ed1541-9d28-454b-913c-210fd8f409ac\" (UID: \"19ed1541-9d28-454b-913c-210fd8f409ac\") "
Nov 03 20:25:14.852892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.852866    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-routing-overlay-volume" pod "19ed1541-9d28-454b-913c-210fd8f409ac" (UID: "19ed1541-9d28-454b-913c-210fd8f409ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:14.853295 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.853272    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-plugins-config-volume" pod "19ed1541-9d28-454b-913c-210fd8f409ac" (UID: "19ed1541-9d28-454b-913c-210fd8f409ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:14.855101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.855077    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-7f8fczlpf"]
Nov 03 20:25:14.855864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.855829    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-provider-credentials" pod "19ed1541-9d28-454b-913c-210fd8f409ac" (UID: "19ed1541-9d28-454b-913c-210fd8f409ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:14.856974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.856424    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-kube-api-access-c9pnf" pod "19ed1541-9d28-454b-913c-210fd8f409ac" (UID: "19ed1541-9d28-454b-913c-210fd8f409ac"). InnerVolumeSpecName "kube-api-access-c9pnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:14.977665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.977594    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:14.977899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.977739    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:14.977899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.977763    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:14.977899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.977808    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:14.977899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.977843    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrgxs\" (UniqueName: \"kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-kube-api-access-vrgxs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:14.978290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.977969    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:14.978290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.978066    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:14.978290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.978158    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:14.984729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.978350    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:14.984729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.978438    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:14.984729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.978457    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ed1541-9d28-454b-913c-210fd8f409ac-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:14.984729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.978471    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:14.984729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.978502    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9pnf\" (UniqueName: \"kubernetes.io/projected/19ed1541-9d28-454b-913c-210fd8f409ac-kube-api-access-c9pnf\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:14.984729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:14.979263    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:15.080309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.080216    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.080309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.080260    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.080588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.080318    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.080588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.080350    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrgxs\" (UniqueName: \"kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-kube-api-access-vrgxs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.080588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.080477    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.080759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.080627    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.080759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.080690    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.080759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.080746    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.080914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.080758    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.080914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.080826    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.080914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.080832    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.081087 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.080916    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.081087 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.081032    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.081464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.081438    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.083065 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.083041    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.083255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.083239    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.092567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.092547    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.092658 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.092616    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrgxs\" (UniqueName: \"kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-kube-api-access-vrgxs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.126095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.126061    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"]
Nov 03 20:25:15.130335 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.130313    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-65d7q2mr9"]
Nov 03 20:25:15.156230 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.156205    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:15.222672 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:25:15.222637    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:25:15.297947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.297878    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n" podUID="ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6" containerName="payload-processing" containerID="cri-o://ea5bc50e35aee4f60757f330ac39aa187c055e6fe48c2fc9de553976d36af76e" gracePeriod=25
Nov 03 20:25:15.471941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.471592    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"]
Nov 03 20:25:15.475717 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:25:15.475659    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c35f36b_9f8c_4dcc_84aa_f3c17b28252e.slice/crio-03ae979361760fc54635284110afd60c1f2ed6ded9e24ef6b1f9aede621a326d WatchSource:0}: Error finding container 03ae979361760fc54635284110afd60c1f2ed6ded9e24ef6b1f9aede621a326d: Status 404 returned error can't find the container with id 03ae979361760fc54635284110afd60c1f2ed6ded9e24ef6b1f9aede621a326d
Nov 03 20:25:15.478811 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.478769    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:25:15.478953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.478859    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:25:15.478953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.478902    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:25:15.651775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.647036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:15.740772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.740378    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:25:15.822528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.821265    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" event={"ID":"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e","Type":"ContainerStarted","Data":"411a7f77513285e0bd3d10efb914958e55485b6921f9300096f36d8817aff58c"}
Nov 03 20:25:15.822528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.821323    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" event={"ID":"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e","Type":"ContainerStarted","Data":"03ae979361760fc54635284110afd60c1f2ed6ded9e24ef6b1f9aede621a326d"}
Nov 03 20:25:15.826433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.826404    5659 generic.go:356] "Generic (PLEG): container finished" podID="ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6" containerID="ea5bc50e35aee4f60757f330ac39aa187c055e6fe48c2fc9de553976d36af76e" exitCode=0
Nov 03 20:25:15.826597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.826459    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n" event={"ID":"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6","Type":"ContainerDied","Data":"ea5bc50e35aee4f60757f330ac39aa187c055e6fe48c2fc9de553976d36af76e"}
Nov 03 20:25:15.826597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.826525    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n" event={"ID":"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6","Type":"ContainerDied","Data":"f05ddeec452b0dfc2e9569bf8d51bb4beba90a3f47b025edb6b371f840566604"}
Nov 03 20:25:15.826597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.826555    5659 scope.go:122] "RemoveContainer" containerID="ea5bc50e35aee4f60757f330ac39aa187c055e6fe48c2fc9de553976d36af76e"
Nov 03 20:25:15.826720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.826637    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"
Nov 03 20:25:15.839540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.839515    5659 scope.go:122] "RemoveContainer" containerID="ea5bc50e35aee4f60757f330ac39aa187c055e6fe48c2fc9de553976d36af76e"
Nov 03 20:25:15.839949 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:25:15.839919    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea5bc50e35aee4f60757f330ac39aa187c055e6fe48c2fc9de553976d36af76e\": container with ID starting with ea5bc50e35aee4f60757f330ac39aa187c055e6fe48c2fc9de553976d36af76e not found: ID does not exist" containerID="ea5bc50e35aee4f60757f330ac39aa187c055e6fe48c2fc9de553976d36af76e"
Nov 03 20:25:15.840031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.839959    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea5bc50e35aee4f60757f330ac39aa187c055e6fe48c2fc9de553976d36af76e"} err="failed to get container status \"ea5bc50e35aee4f60757f330ac39aa187c055e6fe48c2fc9de553976d36af76e\": rpc error: code = NotFound desc = could not find container \"ea5bc50e35aee4f60757f330ac39aa187c055e6fe48c2fc9de553976d36af76e\": container with ID starting with ea5bc50e35aee4f60757f330ac39aa187c055e6fe48c2fc9de553976d36af76e not found: ID does not exist"
Nov 03 20:25:15.854615 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.854561    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" podStartSLOduration=26.854542969 podStartE2EDuration="26.854542969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:15.846323948 +0000 UTC m=+2868.001226370" watchObservedRunningTime="2026-11-03 20:25:15.854542969 +0000 UTC m=+2868.009445391"
Nov 03 20:25:15.855521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.855097    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-kube-api-access-5wcfq\" (UniqueName: \"kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-kube-api-access-5wcfq\") pod \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") "
Nov 03 20:25:15.855651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.855586    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-routing-overlay-volume\") pod \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") "
Nov 03 20:25:15.855985 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.855951    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-routing-overlay-volume" pod "ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6" (UID: "ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:15.856171 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.856154    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-provider-credentials\") pod \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") "
Nov 03 20:25:15.856266 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.856191    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-plugins-config-volume\") pod \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\" (UID: \"ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6\") "
Nov 03 20:25:15.856987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.856964    5659 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-routing-overlay-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:15.857079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.856981    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-plugins-config-volume" pod "ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6" (UID: "ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:15.858819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.858797    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-kube-api-access-5wcfq" pod "ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6" (UID: "ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6"). InnerVolumeSpecName "kube-api-access-5wcfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:15.861336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.861303    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-provider-credentials" pod "ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6" (UID: "ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:15.877452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.877301    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:15.959031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.958997    5659 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-provider-credentials\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:15.959281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.959254    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:15.959281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.959280    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wcfq\" (UniqueName: \"kubernetes.io/projected/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6-kube-api-access-5wcfq\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:15.983511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.983455    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ed1541-9d28-454b-913c-210fd8f409ac" path="/var/lib/kubelet/pods/19ed1541-9d28-454b-913c-210fd8f409ac/volumes"
Nov 03 20:25:15.984323 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.984303    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48f76d2c-0bf7-4ce4-9813-7edf8a1e1791" path="/var/lib/kubelet/pods/48f76d2c-0bf7-4ce4-9813-7edf8a1e1791/volumes"
Nov 03 20:25:15.999630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:15.999604    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:16.046176 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:25:16.046145    5659 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0531f8_3ef6_4c1a_a9cb_d14dd66058a6.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 20:25:16.152731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:16.152685    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"]
Nov 03 20:25:16.163070 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:16.162978    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5f6fwwl6n"]
Nov 03 20:25:16.166731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:16.166704    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:16.167745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:16.167716    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:25:16.167875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:16.167772    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" podUID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 20:25:16.231701 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:16.231607    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:16.361467 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:16.361429    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:16.813505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:16.790359    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:16.967933 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:16.967895    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:17.139042 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:17.138935    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:17.160968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:17.160932    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:25:17.161283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:17.161242    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" podUID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 20:25:17.955100 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:17.955064    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:18.011386 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:18.011324    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6" path="/var/lib/kubelet/pods/ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6/volumes"
Nov 03 20:25:18.159519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:18.156975    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:25:18.159519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:18.157047    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" podUID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 20:25:18.772504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:18.771745    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:19.159564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:19.158938    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:25:19.159564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:19.159005    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" podUID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 20:25:19.289191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:19.289155    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:19.486544 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:19.486514    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:19.620599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:19.620005    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:19.769775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:19.760021    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:20.161560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:20.160808    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:25:20.161560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:20.160873    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" podUID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 20:25:20.194575 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:20.194537    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:20.347494 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:20.347445    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:20.659506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:20.656315    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:20.979514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:20.959305    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:21.110366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:21.110328    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:21.156969 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:21.156928    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:25:21.157255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:21.157222    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" podUID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 20:25:21.539343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:21.539213    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:22.093846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.093728    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:22.192387 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.192352    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"]
Nov 03 20:25:22.192823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.192796    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:25:22.192927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.192873    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" podUID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 20:25:22.194189 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.194164    5659 cpu_manager.go:414] "containerMap: removing container" podUID="ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6" containerName="payload-processing"
Nov 03 20:25:22.194327 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.194315    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6" containerName="payload-processing"
Nov 03 20:25:22.194669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.194652    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0531f8-3ef6-4c1a-a9cb-d14dd66058a6" containerName="payload-processing"
Nov 03 20:25:22.204978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.204956    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"]
Nov 03 20:25:22.205172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.205056    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"
Nov 03 20:25:22.213082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.213058    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d2aa3b-dockercfg-fhd86\""
Nov 03 20:25:22.217123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.217104    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d2aa3b\""
Nov 03 20:25:22.410519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.364349    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:22.477524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.458925    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2tqn\" (UniqueName: \"kubernetes.io/projected/645b8d2f-2118-4686-b386-4e90a7842a08-kube-api-access-x2tqn\") pod \"payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k\" (UID: \"645b8d2f-2118-4686-b386-4e90a7842a08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"
Nov 03 20:25:22.477524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.459111    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b8d2f-2118-4686-b386-4e90a7842a08-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k\" (UID: \"645b8d2f-2118-4686-b386-4e90a7842a08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"
Nov 03 20:25:22.563388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.560862    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b8d2f-2118-4686-b386-4e90a7842a08-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k\" (UID: \"645b8d2f-2118-4686-b386-4e90a7842a08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"
Nov 03 20:25:22.563388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.561280    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tqn\" (UniqueName: \"kubernetes.io/projected/645b8d2f-2118-4686-b386-4e90a7842a08-kube-api-access-x2tqn\") pod \"payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k\" (UID: \"645b8d2f-2118-4686-b386-4e90a7842a08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"
Nov 03 20:25:22.603525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.601037    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:22.614536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.610947    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b8d2f-2118-4686-b386-4e90a7842a08-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k\" (UID: \"645b8d2f-2118-4686-b386-4e90a7842a08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"
Nov 03 20:25:22.644260 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.644219    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2tqn\" (UniqueName: \"kubernetes.io/projected/645b8d2f-2118-4686-b386-4e90a7842a08-kube-api-access-x2tqn\") pod \"payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k\" (UID: \"645b8d2f-2118-4686-b386-4e90a7842a08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"
Nov 03 20:25:22.854876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:22.854835    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"
Nov 03 20:25:23.168059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:23.167957    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:25:23.168059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:23.168028    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" podUID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 20:25:23.271010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:23.270979    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:23.306380 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:25:23.306350    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645b8d2f_2118_4686_b386_4e90a7842a08.slice/crio-04a81202b63f2bddf2920ca1bec55ba31d2ec700f5bb18437f5d91f81c1c53c1 WatchSource:0}: Error finding container 04a81202b63f2bddf2920ca1bec55ba31d2ec700f5bb18437f5d91f81c1c53c1: Status 404 returned error can't find the container with id 04a81202b63f2bddf2920ca1bec55ba31d2ec700f5bb18437f5d91f81c1c53c1
Nov 03 20:25:23.308079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:23.308052    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"]
Nov 03 20:25:24.080871 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:24.080725    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:24.152686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:24.152643    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k" event={"ID":"645b8d2f-2118-4686-b386-4e90a7842a08","Type":"ContainerStarted","Data":"bb85250666cf25e370089e34276c00007af0e6fabc1663b22ae55a1360365b6c"}
Nov 03 20:25:24.152966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:24.152937    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k" event={"ID":"645b8d2f-2118-4686-b386-4e90a7842a08","Type":"ContainerStarted","Data":"04a81202b63f2bddf2920ca1bec55ba31d2ec700f5bb18437f5d91f81c1c53c1"}
Nov 03 20:25:24.154774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:24.154751    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"
Nov 03 20:25:24.464165 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:24.464128    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:24.464652 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:24.464622    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:24.465857 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:24.465837    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:25:24.492283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:24.492233    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k" podStartSLOduration=2.492218348 podStartE2EDuration="2.492218348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:24.187792616 +0000 UTC m=+2876.342695037" watchObservedRunningTime="2026-11-03 20:25:24.492218348 +0000 UTC m=+2876.647120769"
Nov 03 20:25:24.847293 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:24.847206    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:25.626176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:25.626138    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:26.268955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:26.268916    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:27.049134 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:27.049101    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:27.715934 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:27.715899    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:28.362738 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:28.362695    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:29.116564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:29.116528    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:29.750346 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:29.750318    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:30.422156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:30.422125    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:31.138765 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:31.138729    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:31.826129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:31.826093    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:32.603014 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:32.602983    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:33.353870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:33.353836    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:34.130875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:34.130843    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:34.828587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:34.828555    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:35.463762 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:35.463727    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:36.178217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:36.178184    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"
Nov 03 20:25:36.324907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:36.324878    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:37.032782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:37.032746    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:37.712217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:37.712179    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:38.362957 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:38.362921    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:39.028441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:39.028348    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:39.715414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:39.715378    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:40.082139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:40.082050    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:40.242096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:40.242058    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:40.434348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:40.434257    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:41.119676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:41.119640    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:41.249278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:41.249241    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:41.490654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:41.490606    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:41.934066 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:41.933965    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:42.767104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:42.767061    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:43.439267 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:43.439221    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:44.203757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:44.203719    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:45.153649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:45.153613    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:45.274538 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:25:45.274452    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:25:45.933052 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:45.933001    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:46.587603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:46.587567    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:47.454438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:47.454407    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:48.102952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:48.102914    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:48.760498 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:48.760455    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:49.546160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:49.546123    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:50.232939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:50.232905    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:50.960628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:50.960589    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:51.635008 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:51.634968    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:52.301413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:52.301334    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:52.665312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:52.665220    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"]
Nov 03 20:25:52.665559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:52.665532    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" podUID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" containerName="istio-proxy" containerID="cri-o://47211acbfed1630cbff05bbbc8c644eff51e1827e3a9bbb58a8d5c52500cb8b6" gracePeriod=30
Nov 03 20:25:53.130597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:53.130534    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:53.771210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:53.771168    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:54.497917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:54.497886    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:55.243730 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:55.243696    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:56.005759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:56.005727    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:56.504772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:56.504735    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:56.640446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:56.640400    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:56.797524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:56.797411    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:57.446695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:57.446656    5659 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:25:57.446984 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:57.446957    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" podUID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:25:57.447927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:57.447900    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:57.976702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:57.976672    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 20:25:58.012862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.012828    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"]
Nov 03 20:25:58.013817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.013795    5659 cpu_manager.go:414] "containerMap: removing container" podUID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" containerName="istio-proxy"
Nov 03 20:25:58.013888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.013818    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" containerName="istio-proxy"
Nov 03 20:25:58.013963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.013953    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" containerName="istio-proxy"
Nov 03 20:25:58.016971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.016943    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.044028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.042409    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"]
Nov 03 20:25:58.064556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.064510    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-credential-socket\") pod \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") "
Nov 03 20:25:58.064749 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.064575    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-podinfo\") pod \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") "
Nov 03 20:25:58.064749 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.064691    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-kube-api-access-smzvr\" (UniqueName: \"kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-kube-api-access-smzvr\") pod \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") "
Nov 03 20:25:58.064749 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.064719    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-data\") pod \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") "
Nov 03 20:25:58.064943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.064810    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-token\" (UniqueName: \"kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-token\") pod \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") "
Nov 03 20:25:58.064943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.064859    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-certs\") pod \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") "
Nov 03 20:25:58.065064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.064994    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-socket\") pod \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") "
Nov 03 20:25:58.065064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.065046    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-envoy\") pod \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") "
Nov 03 20:25:58.065184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.065151    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istiod-ca-cert\") pod \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\" (UID: \"7ca61f7f-bcc7-4b50-83fc-3b38456848d9\") "
Nov 03 20:25:58.066064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.064805    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-credential-socket" pod "7ca61f7f-bcc7-4b50-83fc-3b38456848d9" (UID: "7ca61f7f-bcc7-4b50-83fc-3b38456848d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.066064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.065398    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-certs" pod "7ca61f7f-bcc7-4b50-83fc-3b38456848d9" (UID: "7ca61f7f-bcc7-4b50-83fc-3b38456848d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.066064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.065547    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.066064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.065605    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.066064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.065636    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.066064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.065657    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-data" pod "7ca61f7f-bcc7-4b50-83fc-3b38456848d9" (UID: "7ca61f7f-bcc7-4b50-83fc-3b38456848d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.066064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.065736    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istiod-ca-cert" pod "7ca61f7f-bcc7-4b50-83fc-3b38456848d9" (UID: "7ca61f7f-bcc7-4b50-83fc-3b38456848d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:58.066064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.065798    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-socket" pod "7ca61f7f-bcc7-4b50-83fc-3b38456848d9" (UID: "7ca61f7f-bcc7-4b50-83fc-3b38456848d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.066064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.065868    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.066064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.066020    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.066750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.066123    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.066750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.066206    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.066750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.066258    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq4g9\" (UniqueName: \"kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-kube-api-access-zq4g9\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.066750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.066324    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.066750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.066497    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.066750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.066518    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.066750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.066532    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.066750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.066546    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.066750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.066560    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.067662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.067624    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-kube-api-access-smzvr" pod "7ca61f7f-bcc7-4b50-83fc-3b38456848d9" (UID: "7ca61f7f-bcc7-4b50-83fc-3b38456848d9"). InnerVolumeSpecName "kube-api-access-smzvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:58.068215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.068187    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-podinfo" pod "7ca61f7f-bcc7-4b50-83fc-3b38456848d9" (UID: "7ca61f7f-bcc7-4b50-83fc-3b38456848d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:25:58.069135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.069111    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-envoy" pod "7ca61f7f-bcc7-4b50-83fc-3b38456848d9" (UID: "7ca61f7f-bcc7-4b50-83fc-3b38456848d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:58.069527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.069470    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-token" pod "7ca61f7f-bcc7-4b50-83fc-3b38456848d9" (UID: "7ca61f7f-bcc7-4b50-83fc-3b38456848d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:58.168504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.168448    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.168723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.168639    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.168723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.168683    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.168845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.168792    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.168910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.168843    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq4g9\" (UniqueName: \"kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-kube-api-access-zq4g9\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.168979 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.168906    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.169054 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.169034    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.169118 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.169082    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.169172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.169112    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.169172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.169152    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.169299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.169270    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.169353 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.169326    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.169411 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.169360    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smzvr\" (UniqueName: \"kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-kube-api-access-smzvr\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.169411 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.169391    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.169411 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.169409    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca61f7f-bcc7-4b50-83fc-3b38456848d9-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:58.169598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.169524    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.169833 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.169810    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.169932 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.169916    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.171800 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.171777    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.171932 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.171813    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.181216 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.181195    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.181367 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.181346    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq4g9\" (UniqueName: \"kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-kube-api-access-zq4g9\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.350237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.349813    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:25:58.418735 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.414001    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:58.591015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.590967    5659 generic.go:356] "Generic (PLEG): container finished" podID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" containerID="47211acbfed1630cbff05bbbc8c644eff51e1827e3a9bbb58a8d5c52500cb8b6" exitCode=0
Nov 03 20:25:58.591221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.591035    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"
Nov 03 20:25:58.591221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.591052    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" event={"ID":"7ca61f7f-bcc7-4b50-83fc-3b38456848d9","Type":"ContainerDied","Data":"47211acbfed1630cbff05bbbc8c644eff51e1827e3a9bbb58a8d5c52500cb8b6"}
Nov 03 20:25:58.591221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.591119    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4" event={"ID":"7ca61f7f-bcc7-4b50-83fc-3b38456848d9","Type":"ContainerDied","Data":"9f10395afeb3978ef8275de8f1c452523c4a5c179e3e6950d796acb9721206ba"}
Nov 03 20:25:58.591221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.591147    5659 scope.go:122] "RemoveContainer" containerID="47211acbfed1630cbff05bbbc8c644eff51e1827e3a9bbb58a8d5c52500cb8b6"
Nov 03 20:25:58.603417 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.603396    5659 scope.go:122] "RemoveContainer" containerID="47211acbfed1630cbff05bbbc8c644eff51e1827e3a9bbb58a8d5c52500cb8b6"
Nov 03 20:25:58.603772 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:25:58.603752    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47211acbfed1630cbff05bbbc8c644eff51e1827e3a9bbb58a8d5c52500cb8b6\": container with ID starting with 47211acbfed1630cbff05bbbc8c644eff51e1827e3a9bbb58a8d5c52500cb8b6 not found: ID does not exist" containerID="47211acbfed1630cbff05bbbc8c644eff51e1827e3a9bbb58a8d5c52500cb8b6"
Nov 03 20:25:58.603839 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.603781    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47211acbfed1630cbff05bbbc8c644eff51e1827e3a9bbb58a8d5c52500cb8b6"} err="failed to get container status \"47211acbfed1630cbff05bbbc8c644eff51e1827e3a9bbb58a8d5c52500cb8b6\": rpc error: code = NotFound desc = could not find container \"47211acbfed1630cbff05bbbc8c644eff51e1827e3a9bbb58a8d5c52500cb8b6\": container with ID starting with 47211acbfed1630cbff05bbbc8c644eff51e1827e3a9bbb58a8d5c52500cb8b6 not found: ID does not exist"
Nov 03 20:25:58.624194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.624167    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"]
Nov 03 20:25:58.628428 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.628404    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-5l7q4"]
Nov 03 20:25:58.726723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.726690    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"]
Nov 03 20:25:58.726968 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:25:58.726936    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76abb9d_37a1_4f23_bd0e_ffd468c5dbe1.slice/crio-60dff2036018da1d7385a424ec683e78ee8b204a06a54a165a648a0253c05928 WatchSource:0}: Error finding container 60dff2036018da1d7385a424ec683e78ee8b204a06a54a165a648a0253c05928: Status 404 returned error can't find the container with id 60dff2036018da1d7385a424ec683e78ee8b204a06a54a165a648a0253c05928
Nov 03 20:25:58.730117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.730085    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:25:58.730202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.730156    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:25:58.730202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:58.730192    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:25:59.136927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:59.130793    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:59.602085 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:59.602045    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f" event={"ID":"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1","Type":"ContainerStarted","Data":"5976e2e2d71acbbf7af0fd68d94b85ed4aeb64c1a30ebc81c14b5ee275d8a052"}
Nov 03 20:25:59.602297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:59.602115    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f" event={"ID":"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1","Type":"ContainerStarted","Data":"60dff2036018da1d7385a424ec683e78ee8b204a06a54a165a648a0253c05928"}
Nov 03 20:25:59.628744 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:59.628691    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f" podStartSLOduration=70.628677222 podStartE2EDuration="1m10.628677222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:59.626829822 +0000 UTC m=+2911.781732273" watchObservedRunningTime="2026-11-03 20:25:59.628677222 +0000 UTC m=+2911.783579642"
Nov 03 20:25:59.871582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:25:59.863632    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:00.002053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:00.001609    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca61f7f-bcc7-4b50-83fc-3b38456848d9" path="/var/lib/kubelet/pods/7ca61f7f-bcc7-4b50-83fc-3b38456848d9/volumes"
Nov 03 20:26:00.363507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:00.356789    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:26:00.366402 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:00.366368    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 20:26:00.366692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:00.366663    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f" podUID="b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 20:26:00.652218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:00.652127    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:01.290199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:01.290165    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:01.362050 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:01.362010    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 20:26:01.364998 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:01.364955    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f" podUID="b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 20:26:02.063946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:02.063611    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:02.351067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:02.350968    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 20:26:02.351389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:02.351357    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f" podUID="b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 20:26:02.738507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:02.738148    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:03.353510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:03.351823    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 20:26:03.353510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:03.351910    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f" podUID="b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 20:26:03.369597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:03.369552    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:04.064030 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:04.063993    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:04.356994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:04.356899    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 20:26:04.357278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:04.357247    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f" podUID="b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 20:26:04.753369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:04.753331    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:05.528708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:05.528670    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:26:05.529721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:05.529697    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:26:05.530815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:05.530794    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:26:05.534801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:05.534776    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:06.247179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:06.247140    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:06.964836 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:06.964801    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:07.778713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:07.778677    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:08.548961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:08.548924    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:09.220042 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:09.220007    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:09.877057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:09.877018    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:10.743039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:10.743006    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:11.361422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:11.361390    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:13.548254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:13.548224    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:14.260456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:14.255144    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:14.898614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:14.898579    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:15.219791 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:26:15.219750    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:26:15.665871 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:15.665781    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:19.372876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:19.372833    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"]
Nov 03 20:26:19.373366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:19.373099    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf" podUID="ab882b17-b18a-4552-9f94-90deb128bbfd" containerName="istio-proxy" containerID="cri-o://684873f468de87420ff2f8d60292121f9185ee114d1e08d0369d76d2870b1990" gracePeriod=30
Nov 03 20:26:20.682011 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:20.681971    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"]
Nov 03 20:26:20.682450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:20.682213    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p" podUID="85068d81-d023-4e25-a6c1-3aff06dc9bd7" containerName="istio-proxy" containerID="cri-o://340408e2cfad879dd92020952cad19999f5f5c2b584427dc487ae6c66b386679" gracePeriod=30
Nov 03 20:26:24.485523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.485462    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:24.692115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.692088    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:26:24.731561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.731527    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"]
Nov 03 20:26:24.732465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.732440    5659 cpu_manager.go:414] "containerMap: removing container" podUID="ab882b17-b18a-4552-9f94-90deb128bbfd" containerName="istio-proxy"
Nov 03 20:26:24.732465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.732465    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab882b17-b18a-4552-9f94-90deb128bbfd" containerName="istio-proxy"
Nov 03 20:26:24.732698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.732684    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab882b17-b18a-4552-9f94-90deb128bbfd" containerName="istio-proxy"
Nov 03 20:26:24.737022 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.736995    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:24.757057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.757021    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"]
Nov 03 20:26:24.798458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.798421    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-token\" (UniqueName: \"kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-token\") pod \"ab882b17-b18a-4552-9f94-90deb128bbfd\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") "
Nov 03 20:26:24.798691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.798516    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-socket\") pod \"ab882b17-b18a-4552-9f94-90deb128bbfd\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") "
Nov 03 20:26:24.798691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.798633    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-podinfo\") pod \"ab882b17-b18a-4552-9f94-90deb128bbfd\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") "
Nov 03 20:26:24.798829 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.798806    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-data\") pod \"ab882b17-b18a-4552-9f94-90deb128bbfd\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") "
Nov 03 20:26:24.798897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.798880    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab882b17-b18a-4552-9f94-90deb128bbfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab882b17-b18a-4552-9f94-90deb128bbfd-istiod-ca-cert\") pod \"ab882b17-b18a-4552-9f94-90deb128bbfd\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") "
Nov 03 20:26:24.799021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.799002    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-envoy\") pod \"ab882b17-b18a-4552-9f94-90deb128bbfd\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") "
Nov 03 20:26:24.799697 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.799667    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-credential-socket\") pod \"ab882b17-b18a-4552-9f94-90deb128bbfd\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") "
Nov 03 20:26:24.800225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.800206    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-kube-api-access-wplwn\" (UniqueName: \"kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-kube-api-access-wplwn\") pod \"ab882b17-b18a-4552-9f94-90deb128bbfd\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") "
Nov 03 20:26:24.800750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.800729    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-certs\") pod \"ab882b17-b18a-4552-9f94-90deb128bbfd\" (UID: \"ab882b17-b18a-4552-9f94-90deb128bbfd\") "
Nov 03 20:26:24.801139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.799176    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-data" pod "ab882b17-b18a-4552-9f94-90deb128bbfd" (UID: "ab882b17-b18a-4552-9f94-90deb128bbfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:24.801242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.799180    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-socket" pod "ab882b17-b18a-4552-9f94-90deb128bbfd" (UID: "ab882b17-b18a-4552-9f94-90deb128bbfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:24.801242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.799333    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab882b17-b18a-4552-9f94-90deb128bbfd-istiod-ca-cert" pod "ab882b17-b18a-4552-9f94-90deb128bbfd" (UID: "ab882b17-b18a-4552-9f94-90deb128bbfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:24.801242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.800056    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-credential-socket" pod "ab882b17-b18a-4552-9f94-90deb128bbfd" (UID: "ab882b17-b18a-4552-9f94-90deb128bbfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:24.801242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.801042    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-certs" pod "ab882b17-b18a-4552-9f94-90deb128bbfd" (UID: "ab882b17-b18a-4552-9f94-90deb128bbfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:24.802362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.802335    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:24.802470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.802367    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab882b17-b18a-4552-9f94-90deb128bbfd-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:24.802470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.802383    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:24.802470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.802398    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:24.802470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.802413    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:24.803898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.803858    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-podinfo" pod "ab882b17-b18a-4552-9f94-90deb128bbfd" (UID: "ab882b17-b18a-4552-9f94-90deb128bbfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:24.804052 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.803955    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-envoy" pod "ab882b17-b18a-4552-9f94-90deb128bbfd" (UID: "ab882b17-b18a-4552-9f94-90deb128bbfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:24.804052 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.803981    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-token" pod "ab882b17-b18a-4552-9f94-90deb128bbfd" (UID: "ab882b17-b18a-4552-9f94-90deb128bbfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:24.804318 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.804297    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-kube-api-access-wplwn" pod "ab882b17-b18a-4552-9f94-90deb128bbfd" (UID: "ab882b17-b18a-4552-9f94-90deb128bbfd"). InnerVolumeSpecName "kube-api-access-wplwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:24.897021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.896926    5659 generic.go:356] "Generic (PLEG): container finished" podID="ab882b17-b18a-4552-9f94-90deb128bbfd" containerID="684873f468de87420ff2f8d60292121f9185ee114d1e08d0369d76d2870b1990" exitCode=0
Nov 03 20:26:24.897021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.896971    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"
Nov 03 20:26:24.897021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.897005    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf" event={"ID":"ab882b17-b18a-4552-9f94-90deb128bbfd","Type":"ContainerDied","Data":"684873f468de87420ff2f8d60292121f9185ee114d1e08d0369d76d2870b1990"}
Nov 03 20:26:24.897273 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.897062    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf" event={"ID":"ab882b17-b18a-4552-9f94-90deb128bbfd","Type":"ContainerDied","Data":"3b6cfc9918f1fe5082f71a879bb67ca8678f7b672b4bef4d8e8e019e370bffee"}
Nov 03 20:26:24.897273 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.897088    5659 scope.go:122] "RemoveContainer" containerID="684873f468de87420ff2f8d60292121f9185ee114d1e08d0369d76d2870b1990"
Nov 03 20:26:24.903936 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.903905    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:24.904081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.904048    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:24.904273 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.904246    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:24.904364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.904346    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:24.904421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.904391    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd9vc\" (UniqueName: \"kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-kube-api-access-vd9vc\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:24.904499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.904420    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:24.904499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.904444    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:24.904627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.904605    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:24.904724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.904707    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:24.904959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.904937    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wplwn\" (UniqueName: \"kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-kube-api-access-wplwn\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:24.905048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.904965    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:24.905048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.904980    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:24.905048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.904995    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab882b17-b18a-4552-9f94-90deb128bbfd-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:24.910053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.910027    5659 scope.go:122] "RemoveContainer" containerID="684873f468de87420ff2f8d60292121f9185ee114d1e08d0369d76d2870b1990"
Nov 03 20:26:24.910364 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:26:24.910339    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684873f468de87420ff2f8d60292121f9185ee114d1e08d0369d76d2870b1990\": container with ID starting with 684873f468de87420ff2f8d60292121f9185ee114d1e08d0369d76d2870b1990 not found: ID does not exist" containerID="684873f468de87420ff2f8d60292121f9185ee114d1e08d0369d76d2870b1990"
Nov 03 20:26:24.910429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.910374    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684873f468de87420ff2f8d60292121f9185ee114d1e08d0369d76d2870b1990"} err="failed to get container status \"684873f468de87420ff2f8d60292121f9185ee114d1e08d0369d76d2870b1990\": rpc error: code = NotFound desc = could not find container \"684873f468de87420ff2f8d60292121f9185ee114d1e08d0369d76d2870b1990\": container with ID starting with 684873f468de87420ff2f8d60292121f9185ee114d1e08d0369d76d2870b1990 not found: ID does not exist"
Nov 03 20:26:24.924437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.924407    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"]
Nov 03 20:26:24.929177 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:24.929151    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-wjpxf"]
Nov 03 20:26:25.006278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.006230    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.006525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.006365    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.006708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.006558    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.006708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.006632    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.006708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.006663    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd9vc\" (UniqueName: \"kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-kube-api-access-vd9vc\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.006708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.006699    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.006942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.006735    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.006942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.006792    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.006942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.006858    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.007913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.007888    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.008040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.007963    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.008114 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.008048    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.008844 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.008815    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.010547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.010523    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.012566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.012445    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.014148 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.014128    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.025970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.025938    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd9vc\" (UniqueName: \"kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-kube-api-access-vd9vc\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.026204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.026171    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.058563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.058468    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:25.375922 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:26:25.375888    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0fa848_0fe6_41d8_b068_8bd42f81bb6e.slice/crio-07b6cdde9cce4df8b8524aadf057c6db70876adee741f339fd9264cb99bbea06 WatchSource:0}: Error finding container 07b6cdde9cce4df8b8524aadf057c6db70876adee741f339fd9264cb99bbea06: Status 404 returned error can't find the container with id 07b6cdde9cce4df8b8524aadf057c6db70876adee741f339fd9264cb99bbea06
Nov 03 20:26:25.378059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.378026    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"]
Nov 03 20:26:25.379350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.379310    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:26:25.379450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.379384    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:26:25.379450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.379414    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:26:25.912720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.912682    5659 generic.go:356] "Generic (PLEG): container finished" podID="85068d81-d023-4e25-a6c1-3aff06dc9bd7" containerID="340408e2cfad879dd92020952cad19999f5f5c2b584427dc487ae6c66b386679" exitCode=0
Nov 03 20:26:25.913221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.912781    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p" event={"ID":"85068d81-d023-4e25-a6c1-3aff06dc9bd7","Type":"ContainerDied","Data":"340408e2cfad879dd92020952cad19999f5f5c2b584427dc487ae6c66b386679"}
Nov 03 20:26:25.917520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.916839    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" event={"ID":"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e","Type":"ContainerStarted","Data":"df64b065a07a619c7cf3705172be7c7543e8d9685ba16888bcb9054792945514"}
Nov 03 20:26:25.917520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.916900    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" event={"ID":"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e","Type":"ContainerStarted","Data":"07b6cdde9cce4df8b8524aadf057c6db70876adee741f339fd9264cb99bbea06"}
Nov 03 20:26:25.956554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.952352    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:25.971846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.971814    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab882b17-b18a-4552-9f94-90deb128bbfd" path="/var/lib/kubelet/pods/ab882b17-b18a-4552-9f94-90deb128bbfd/volumes"
Nov 03 20:26:25.990286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:25.990261    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 20:26:26.016285 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.016222    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podStartSLOduration=97.016207336 podStartE2EDuration="1m37.016207336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:25.951647273 +0000 UTC m=+2938.106549694" watchObservedRunningTime="2026-11-03 20:26:26.016207336 +0000 UTC m=+2938.171109754"
Nov 03 20:26:26.028028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.027980    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"]
Nov 03 20:26:26.029213 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.029190    5659 cpu_manager.go:414] "containerMap: removing container" podUID="85068d81-d023-4e25-a6c1-3aff06dc9bd7" containerName="istio-proxy"
Nov 03 20:26:26.029280 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.029217    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85068d81-d023-4e25-a6c1-3aff06dc9bd7" containerName="istio-proxy"
Nov 03 20:26:26.029477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.029462    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="85068d81-d023-4e25-a6c1-3aff06dc9bd7" containerName="istio-proxy"
Nov 03 20:26:26.030448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.030421    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-credential-socket\") pod \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") "
Nov 03 20:26:26.030595 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.030471    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-token\" (UniqueName: \"kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-token\") pod \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") "
Nov 03 20:26:26.030660 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.030636    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istiod-ca-cert\") pod \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") "
Nov 03 20:26:26.030724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.030712    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-certs\") pod \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") "
Nov 03 20:26:26.030779 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.030755    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-credential-socket" pod "85068d81-d023-4e25-a6c1-3aff06dc9bd7" (UID: "85068d81-d023-4e25-a6c1-3aff06dc9bd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.030836 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.030795    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-socket\") pod \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") "
Nov 03 20:26:26.030891 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.030837    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-kube-api-access-26jjx\" (UniqueName: \"kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-kube-api-access-26jjx\") pod \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") "
Nov 03 20:26:26.031328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.031026    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-certs" pod "85068d81-d023-4e25-a6c1-3aff06dc9bd7" (UID: "85068d81-d023-4e25-a6c1-3aff06dc9bd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.031328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.031037    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-data\") pod \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") "
Nov 03 20:26:26.031328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.031060    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-socket" pod "85068d81-d023-4e25-a6c1-3aff06dc9bd7" (UID: "85068d81-d023-4e25-a6c1-3aff06dc9bd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.031328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.031080    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istiod-ca-cert" pod "85068d81-d023-4e25-a6c1-3aff06dc9bd7" (UID: "85068d81-d023-4e25-a6c1-3aff06dc9bd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:26.031328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.031188    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-envoy\") pod \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") "
Nov 03 20:26:26.031328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.031287    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-data" pod "85068d81-d023-4e25-a6c1-3aff06dc9bd7" (UID: "85068d81-d023-4e25-a6c1-3aff06dc9bd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.031328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.031296    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-podinfo\") pod \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\" (UID: \"85068d81-d023-4e25-a6c1-3aff06dc9bd7\") "
Nov 03 20:26:26.032234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.032209    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.032333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.032241    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.032333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.032258    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.032333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.032272    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.032333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.032285    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.033872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.033846    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-token" pod "85068d81-d023-4e25-a6c1-3aff06dc9bd7" (UID: "85068d81-d023-4e25-a6c1-3aff06dc9bd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:26.034434 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.034411    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-kube-api-access-26jjx" pod "85068d81-d023-4e25-a6c1-3aff06dc9bd7" (UID: "85068d81-d023-4e25-a6c1-3aff06dc9bd7"). InnerVolumeSpecName "kube-api-access-26jjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:26.041990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.039957    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-podinfo" pod "85068d81-d023-4e25-a6c1-3aff06dc9bd7" (UID: "85068d81-d023-4e25-a6c1-3aff06dc9bd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:26.050440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.050414    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"]
Nov 03 20:26:26.050677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.050515    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.057872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.057575    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d2aa3b-serving-cert\""
Nov 03 20:26:26.061272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.061252    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:26.062178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.062031    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:26.062178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.062085    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:26.068534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.068380    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-envoy" pod "85068d81-d023-4e25-a6c1-3aff06dc9bd7" (UID: "85068d81-d023-4e25-a6c1-3aff06dc9bd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.134986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.134906    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-metrics-certs\") pod \"maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.136006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.135979    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwsd6\" (UniqueName: \"kubernetes.io/projected/59b60f28-7800-45ba-9fa6-3964633c5df3-kube-api-access-kwsd6\") pod \"maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.136241 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.136200    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-maas-api-tls\") pod \"maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.136396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.136375    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b60f28-7800-45ba-9fa6-3964633c5df3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.136721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.136699    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.136721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.136720    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.136867 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.136730    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.136867 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.136742    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26jjx\" (UniqueName: \"kubernetes.io/projected/85068d81-d023-4e25-a6c1-3aff06dc9bd7-kube-api-access-26jjx\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.239029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.238938    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwsd6\" (UniqueName: \"kubernetes.io/projected/59b60f28-7800-45ba-9fa6-3964633c5df3-kube-api-access-kwsd6\") pod \"maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.239029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.238992    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-maas-api-tls\") pod \"maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.239247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.239037    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b60f28-7800-45ba-9fa6-3964633c5df3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.239247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.239236    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-metrics-certs\") pod \"maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.239829 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.239798    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b60f28-7800-45ba-9fa6-3964633c5df3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.242234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.242213    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-metrics-certs\") pod \"maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.242378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.242356    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-maas-api-tls\") pod \"maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.251165 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.251125    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwsd6\" (UniqueName: \"kubernetes.io/projected/59b60f28-7800-45ba-9fa6-3964633c5df3-kube-api-access-kwsd6\") pod \"maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.377876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.377834    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:26.406801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.406757    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:26.780931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.780904    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"]
Nov 03 20:26:26.781712 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:26:26.781681    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b60f28_7800_45ba_9fa6_3964633c5df3.slice/crio-156cce8c215182bafa50a7aa6f6a5f50bf10b426db96bdb88172ceeb1c130836 WatchSource:0}: Error finding container 156cce8c215182bafa50a7aa6f6a5f50bf10b426db96bdb88172ceeb1c130836: Status 404 returned error can't find the container with id 156cce8c215182bafa50a7aa6f6a5f50bf10b426db96bdb88172ceeb1c130836
Nov 03 20:26:26.783778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.783758    5659 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:26:26.927859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.927778    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" event={"ID":"59b60f28-7800-45ba-9fa6-3964633c5df3","Type":"ContainerStarted","Data":"156cce8c215182bafa50a7aa6f6a5f50bf10b426db96bdb88172ceeb1c130836"}
Nov 03 20:26:26.929290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.929271    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"
Nov 03 20:26:26.929415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.929311    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p" event={"ID":"85068d81-d023-4e25-a6c1-3aff06dc9bd7","Type":"ContainerDied","Data":"85f4dd7ed00c14d6a870c6ad4c97fc5ebcdb2fee8bb99cc45e876ac7f27a13c0"}
Nov 03 20:26:26.929415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.929365    5659 scope.go:122] "RemoveContainer" containerID="340408e2cfad879dd92020952cad19999f5f5c2b584427dc487ae6c66b386679"
Nov 03 20:26:26.959904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.959877    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"]
Nov 03 20:26:26.970419 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:26.970379    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-g8w8p"]
Nov 03 20:26:27.059447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:27.059328    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:27.059447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:27.059398    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:27.144680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:27.144642    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:27.917506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:27.910748    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:28.030506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:28.025295    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podStartSLOduration=46.444972708 podStartE2EDuration="47.025276598s" totalImagesPullingTime="580.30389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:28.02285274 +0000 UTC m=+2940.177755159" watchObservedRunningTime="2026-11-03 20:26:28.025276598 +0000 UTC m=+2940.180179018"
Nov 03 20:26:28.070475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:28.070339    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:28.070475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:28.070391    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:28.076797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:28.071045    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85068d81-d023-4e25-a6c1-3aff06dc9bd7" path="/var/lib/kubelet/pods/85068d81-d023-4e25-a6c1-3aff06dc9bd7/volumes"
Nov 03 20:26:28.076797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:28.071545    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" event={"ID":"59b60f28-7800-45ba-9fa6-3964633c5df3","Type":"ContainerStarted","Data":"8dc51c52507b3abd5b29387b230d0ca0bd06daecb9ffa9ffbc3a1618b6402f66"}
Nov 03 20:26:28.076797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:28.071608    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:26:28.732523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:28.732464    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:29.059320 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:29.059231    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:29.059320 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:29.059291    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:29.531410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:29.531369    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:30.059801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:30.059760    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:30.060223 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:30.059820    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:30.163011 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:30.162975    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"]
Nov 03 20:26:30.163195 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:30.163174    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs" podUID="2477710a-1350-4fa2-9381-b2e6c4fcb6b1" containerName="istio-proxy" containerID="cri-o://423aaa21382ab1445b77bee899471fd72d3834e5d35faaf174ae80d5574871bf" gracePeriod=30
Nov 03 20:26:30.249194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:30.249125    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:30.957398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:30.957359    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:31.059282 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:31.059237    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:31.059464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:31.059303    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:31.756685 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:31.756652    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:32.059790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:32.059686    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:32.059790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:32.059755    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:32.683136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:32.678396    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:33.059551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:33.059514    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:33.060064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:33.059567    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:33.375362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:33.375263    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:34.034560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:34.033616    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:26:34.061514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:34.059703    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:34.061514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:34.059760    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:34.356568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:34.356446    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:35.061941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.061897    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:35.062568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.062513    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:35.085767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.085736    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:35.572111 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.570933    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:26:35.618788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.618745    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"]
Nov 03 20:26:35.620902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.620868    5659 cpu_manager.go:414] "containerMap: removing container" podUID="2477710a-1350-4fa2-9381-b2e6c4fcb6b1" containerName="istio-proxy"
Nov 03 20:26:35.621090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.621075    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2477710a-1350-4fa2-9381-b2e6c4fcb6b1" containerName="istio-proxy"
Nov 03 20:26:35.621666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.621603    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="2477710a-1350-4fa2-9381-b2e6c4fcb6b1" containerName="istio-proxy"
Nov 03 20:26:35.641077 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.641036    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-kube-api-access-5s8tf\" (UniqueName: \"kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-kube-api-access-5s8tf\") pod \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") "
Nov 03 20:26:35.641249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.641172    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-podinfo\") pod \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") "
Nov 03 20:26:35.641249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.641220    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-certs\") pod \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") "
Nov 03 20:26:35.641389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.641285    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-token\" (UniqueName: \"kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-token\") pod \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") "
Nov 03 20:26:35.641448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.641433    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-socket\") pod \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") "
Nov 03 20:26:35.641526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.641471    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istiod-ca-cert\") pod \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") "
Nov 03 20:26:35.641593 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.641522    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-credential-socket\") pod \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") "
Nov 03 20:26:35.641652 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.641619    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-envoy\") pod \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") "
Nov 03 20:26:35.641866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.641847    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-data\") pod \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\" (UID: \"2477710a-1350-4fa2-9381-b2e6c4fcb6b1\") "
Nov 03 20:26:35.643620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.643590    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-data" pod "2477710a-1350-4fa2-9381-b2e6c4fcb6b1" (UID: "2477710a-1350-4fa2-9381-b2e6c4fcb6b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.648813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.648785    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-socket" pod "2477710a-1350-4fa2-9381-b2e6c4fcb6b1" (UID: "2477710a-1350-4fa2-9381-b2e6c4fcb6b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.649292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.649272    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istiod-ca-cert" pod "2477710a-1350-4fa2-9381-b2e6c4fcb6b1" (UID: "2477710a-1350-4fa2-9381-b2e6c4fcb6b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:35.649665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.649647    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-credential-socket" pod "2477710a-1350-4fa2-9381-b2e6c4fcb6b1" (UID: "2477710a-1350-4fa2-9381-b2e6c4fcb6b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.683937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.683895    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-token" pod "2477710a-1350-4fa2-9381-b2e6c4fcb6b1" (UID: "2477710a-1350-4fa2-9381-b2e6c4fcb6b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.687936 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.687896    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-certs" pod "2477710a-1350-4fa2-9381-b2e6c4fcb6b1" (UID: "2477710a-1350-4fa2-9381-b2e6c4fcb6b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.691732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.689326    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-podinfo" pod "2477710a-1350-4fa2-9381-b2e6c4fcb6b1" (UID: "2477710a-1350-4fa2-9381-b2e6c4fcb6b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:35.700574 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.699244    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-envoy" pod "2477710a-1350-4fa2-9381-b2e6c4fcb6b1" (UID: "2477710a-1350-4fa2-9381-b2e6c4fcb6b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.706522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.702256    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-kube-api-access-5s8tf" pod "2477710a-1350-4fa2-9381-b2e6c4fcb6b1" (UID: "2477710a-1350-4fa2-9381-b2e6c4fcb6b1"). InnerVolumeSpecName "kube-api-access-5s8tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.706522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.702294    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"]
Nov 03 20:26:35.706522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.702410    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.725147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.712928    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-28b78c-openshift-default-dockercfg-s5mc7\""
Nov 03 20:26:35.760514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.759235    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.760514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.759302    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s8tf\" (UniqueName: \"kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-kube-api-access-5s8tf\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.760514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.759323    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.760514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.759337    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.760514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.759375    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.760514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.759391    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.760514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.759405    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.760514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.759437    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.760514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.759455    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2477710a-1350-4fa2-9381-b2e6c4fcb6b1-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.866083 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.866014    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-credential-socket\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.866290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.866158    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-token\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.866387 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.866364    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df7dq\" (UniqueName: \"kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-kube-api-access-df7dq\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.866511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.866448    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-certs\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.866594 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.866546    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-podinfo\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.867445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.866740    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0a991c-5faa-416f-be22-d7b258a358e1-istiod-ca-cert\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.867445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.866840    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-socket\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.867445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.867032    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-data\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.867445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.867109    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-envoy\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.969058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.969021    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-certs\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.969058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.969056    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-podinfo\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.969343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.969146    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0a991c-5faa-416f-be22-d7b258a358e1-istiod-ca-cert\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.969343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.969196    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-socket\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.969343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.969320    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-data\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.969578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.969351    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-envoy\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.969578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.969461    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-credential-socket\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.969578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.969503    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-token\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.969578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.969513    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-certs\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.969791 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.969675    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df7dq\" (UniqueName: \"kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-kube-api-access-df7dq\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.970206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.969908    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-data\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.970206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.970177    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-socket\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.970863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.970565    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0a991c-5faa-416f-be22-d7b258a358e1-istiod-ca-cert\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.970863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.970719    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-credential-socket\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.971874 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.971851    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-podinfo\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.972350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.972330    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-envoy\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.982084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.982068    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df7dq\" (UniqueName: \"kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-kube-api-access-df7dq\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:35.982197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:35.982179    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-token\") pod \"e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") " pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:36.061985 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.061941    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:36.062557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.062026    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:36.085440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.085412    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:36.229827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.229744    5659 generic.go:356] "Generic (PLEG): container finished" podID="2477710a-1350-4fa2-9381-b2e6c4fcb6b1" containerID="423aaa21382ab1445b77bee899471fd72d3834e5d35faaf174ae80d5574871bf" exitCode=0
Nov 03 20:26:36.229968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.229864    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs" event={"ID":"2477710a-1350-4fa2-9381-b2e6c4fcb6b1","Type":"ContainerDied","Data":"423aaa21382ab1445b77bee899471fd72d3834e5d35faaf174ae80d5574871bf"}
Nov 03 20:26:36.229968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.229916    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs" event={"ID":"2477710a-1350-4fa2-9381-b2e6c4fcb6b1","Type":"ContainerDied","Data":"63d546880b4cf72f0ce1379e646a3c398ab283784efe688b98c35659eca2d36c"}
Nov 03 20:26:36.229968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.229944    5659 scope.go:122] "RemoveContainer" containerID="423aaa21382ab1445b77bee899471fd72d3834e5d35faaf174ae80d5574871bf"
Nov 03 20:26:36.230140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.230126    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"
Nov 03 20:26:36.300110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.298650    5659 scope.go:122] "RemoveContainer" containerID="423aaa21382ab1445b77bee899471fd72d3834e5d35faaf174ae80d5574871bf"
Nov 03 20:26:36.307444 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:26:36.307323    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423aaa21382ab1445b77bee899471fd72d3834e5d35faaf174ae80d5574871bf\": container with ID starting with 423aaa21382ab1445b77bee899471fd72d3834e5d35faaf174ae80d5574871bf not found: ID does not exist" containerID="423aaa21382ab1445b77bee899471fd72d3834e5d35faaf174ae80d5574871bf"
Nov 03 20:26:36.307444 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.307369    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423aaa21382ab1445b77bee899471fd72d3834e5d35faaf174ae80d5574871bf"} err="failed to get container status \"423aaa21382ab1445b77bee899471fd72d3834e5d35faaf174ae80d5574871bf\": rpc error: code = NotFound desc = could not find container \"423aaa21382ab1445b77bee899471fd72d3834e5d35faaf174ae80d5574871bf\": container with ID starting with 423aaa21382ab1445b77bee899471fd72d3834e5d35faaf174ae80d5574871bf not found: ID does not exist"
Nov 03 20:26:36.312946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.312914    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"]
Nov 03 20:26:36.317374 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.317347    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xvlbs"]
Nov 03 20:26:36.430294 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:26:36.430259    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0a991c_5faa_416f_be22_d7b258a358e1.slice/crio-a196bb131059ccb0cba2e133af3913ca1346b7912c123fb6ffd2e66cbab307e1 WatchSource:0}: Error finding container a196bb131059ccb0cba2e133af3913ca1346b7912c123fb6ffd2e66cbab307e1: Status 404 returned error can't find the container with id a196bb131059ccb0cba2e133af3913ca1346b7912c123fb6ffd2e66cbab307e1
Nov 03 20:26:36.430439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.430398    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"]
Nov 03 20:26:36.434048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.433999    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:26:36.434191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.434099    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:26:36.434191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.434148    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:26:36.881407 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:36.881312    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:37.062503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:37.062028    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:37.062503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:37.062085    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:37.168508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:37.168406    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:37.300287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:37.300232    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" event={"ID":"4d0a991c-5faa-416f-be22-d7b258a358e1","Type":"ContainerStarted","Data":"d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de"}
Nov 03 20:26:37.300620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:37.300602    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" event={"ID":"4d0a991c-5faa-416f-be22-d7b258a358e1","Type":"ContainerStarted","Data":"a196bb131059ccb0cba2e133af3913ca1346b7912c123fb6ffd2e66cbab307e1"}
Nov 03 20:26:37.359508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:37.357314    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podStartSLOduration=200.357292374 podStartE2EDuration="3m20.357292374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:37.326129552 +0000 UTC m=+2949.481031972" watchObservedRunningTime="2026-11-03 20:26:37.357292374 +0000 UTC m=+2949.512194794"
Nov 03 20:26:37.975386 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:37.975349    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2477710a-1350-4fa2-9381-b2e6c4fcb6b1" path="/var/lib/kubelet/pods/2477710a-1350-4fa2-9381-b2e6c4fcb6b1/volumes"
Nov 03 20:26:38.059881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:38.059841    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:38.060154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:38.060127    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:38.095912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:38.095878    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:38.097637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:38.097608    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:38.097855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:38.097824    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:38.121567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:38.121535    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:38.856776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:38.856711    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:39.019738 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:39.019701    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:26:39.063590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:39.059860    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:39.063590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:39.059926    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:39.086318 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:39.086266    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:39.086516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:39.086337    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:39.645568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:39.645531    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:40.066505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:40.065599    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:40.068748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:40.067189    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:26:40.068748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:40.067244    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:26:40.085944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:40.085906    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:40.086114 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:40.085959    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:40.315997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:40.315949    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:40.961859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:40.961820    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:41.067037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:41.067007    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:41.069000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:41.068973    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:26:41.086283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:41.086244    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:41.086432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:41.086305    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:41.679314 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:41.679268    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:42.085811 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:42.085776    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:42.086220 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:42.085827    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:42.405163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:42.395396    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:43.086280 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:43.086236    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:43.086717 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:43.086301    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:43.189150 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:43.189117    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:44.020080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:44.020045    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:26:44.085955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:44.085924    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:44.086127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:44.085977    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:44.274608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:44.271863    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:44.931292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:44.931260    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:45.086043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:45.086001    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:45.086253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:45.086070    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:45.222873 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:26:45.222833    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:26:45.759994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:45.759958    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:46.086217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:46.086131    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:46.086217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:46.086183    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:46.542051 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:46.542014    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:47.085884 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:47.085841    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:47.086356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:47.085902    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:47.238046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:47.238011    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:48.058617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:48.058584    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:48.090928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:48.090890    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:48.091614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:48.091578    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:48.845147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:48.845113    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:49.019274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:49.019220    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:26:49.086678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:49.086641    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:49.086847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:49.086692    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:49.721973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:49.721940    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:50.085827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:50.085751    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:50.085827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:50.085800    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:50.450976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:50.450877    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:51.086166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:51.086130    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:26:51.086596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:51.086404    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:51.086596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:51.086460    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:51.302700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:51.302663    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:51.939179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:51.939133    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:52.086132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:52.086085    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:52.086366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:52.086148    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:52.652807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:52.652773    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:53.085988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:53.085952    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:53.086181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:53.086031    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:53.337561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:53.337443    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:54.020111 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:54.020065    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:26:54.085865 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:54.085830    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:54.086056 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:54.085886    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:54.325884 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:54.325793    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:54.996805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:54.996767    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:55.085770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:55.085736    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:55.085939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:55.085797    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:55.745743 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:55.745692    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:56.086526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:56.086397    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:56.086526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:56.086466    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:56.441976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:56.441892    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:57.086353 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:57.086309    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:57.086593 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:57.086374    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:57.149694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:57.149650    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:57.851987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:57.851917    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:58.086355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:58.086318    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:58.086579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:58.086380    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:58.531783 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:58.531749    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:59.020259 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:59.020222    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:26:59.085878 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:59.085843    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:59.086075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:59.085911    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:59.161687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:59.161644    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:59.955452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:26:59.955411    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:00.086815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:00.086780    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:27:00.087037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:00.086853    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:27:00.768496 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:00.768459    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:01.086048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:01.085957    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:27:01.086048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:01.086030    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:27:01.600084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:01.600042    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:02.086644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:02.086612    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:27:02.087041 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:02.086677    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:27:02.504332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:02.504292    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:03.086804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:03.086767    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:27:03.090178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:03.090144    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:27:03.657666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:03.657585    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:03.809231 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:03.809192    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:04.019392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:04.019356    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:04.019659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:04.019527    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:27:04.087914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:04.087866    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:27:04.088348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:04.087951    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:27:05.086532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:05.086471    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:27:05.086746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:05.086546    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:27:05.311919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:05.311840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:06.086010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:06.085973    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:27:06.086238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:06.086032    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:27:06.351556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:06.351364    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:06.378275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:06.378233    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:27:07.085767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:07.085734    5659 patch_prober.go:29] interesting pod/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:27:07.085951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:07.085803    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:27:07.085951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:07.085845    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:27:07.086524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:07.086497    5659 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de"} pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" containerMessage="Container istio-proxy failed startup probe, will be restarted"
Nov 03 20:27:07.086642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:07.086552    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" containerID="cri-o://d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de" gracePeriod=30
Nov 03 20:27:07.260575 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:07.257142    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:08.244576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:08.244540    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:09.019886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:09.019802    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:27:09.217957 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:09.214030    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:10.030337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:10.028818    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:10.871389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:10.871315    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:11.688914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:11.688877    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:12.206070 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:12.206034    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:27:12.206178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:12.206117    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:27:12.206178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:12.206160    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:27:12.634527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:12.634223    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:12.792284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:12.792250    5659 generic.go:356] "Generic (PLEG): container finished" podID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerID="d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de" exitCode=0
Nov 03 20:27:12.792804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:12.792328    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" event={"ID":"4d0a991c-5faa-416f-be22-d7b258a358e1","Type":"ContainerDied","Data":"d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de"}
Nov 03 20:27:12.792804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:12.792386    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" event={"ID":"4d0a991c-5faa-416f-be22-d7b258a358e1","Type":"ContainerStarted","Data":"1cdef80f2f136980bda4e3d8a25fc85e80c5934887110c2bd9abe9fda27fd3b4"}
Nov 03 20:27:13.085985 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:13.085955    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:27:13.091136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:13.091111    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:27:13.581210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:13.581173    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:13.800123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:13.800088    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:27:13.801199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:13.801178    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:27:14.019554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:14.019460    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:27:14.412355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:14.412277    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:14.542957 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:14.542920    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:15.219290 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:15.219245    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:27:15.503858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:15.503721    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:16.301664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:16.294254    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:16.378942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:16.378893    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:27:16.947205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:16.947169    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:17.723544 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:17.723472    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:18.406656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:18.406620    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:19.020236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:19.020197    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:27:19.126957 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:19.126925    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:20.058983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:20.058944    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:20.829255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:20.826963    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:20.915591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:20.915561    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"]
Nov 03 20:27:20.929895 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:20.929866    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"]
Nov 03 20:27:20.930156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:20.930143    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"
Nov 03 20:27:20.935860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:20.935827    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:27:20.937162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:20.937137    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\""
Nov 03 20:27:20.997862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:20.997825    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-xmsnl\" (UID: \"6f040abf-425e-4d21-9015-489f3dd38ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"
Nov 03 20:27:20.998246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:20.998221    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrr5\" (UniqueName: \"kubernetes.io/projected/6f040abf-425e-4d21-9015-489f3dd38ec5-kube-api-access-vzrr5\") pod \"maas-api-revoke-sub-abad4031-xmsnl\" (UID: \"6f040abf-425e-4d21-9015-489f3dd38ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"
Nov 03 20:27:20.998339 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:20.998327    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-xmsnl\" (UID: \"6f040abf-425e-4d21-9015-489f3dd38ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"
Nov 03 20:27:21.100572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:21.100467    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrr5\" (UniqueName: \"kubernetes.io/projected/6f040abf-425e-4d21-9015-489f3dd38ec5-kube-api-access-vzrr5\") pod \"maas-api-revoke-sub-abad4031-xmsnl\" (UID: \"6f040abf-425e-4d21-9015-489f3dd38ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"
Nov 03 20:27:21.101001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:21.100577    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-xmsnl\" (UID: \"6f040abf-425e-4d21-9015-489f3dd38ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"
Nov 03 20:27:21.101001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:21.100699    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-xmsnl\" (UID: \"6f040abf-425e-4d21-9015-489f3dd38ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"
Nov 03 20:27:21.101393 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:21.101371    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-xmsnl\" (UID: \"6f040abf-425e-4d21-9015-489f3dd38ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"
Nov 03 20:27:21.103475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:21.103450    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-xmsnl\" (UID: \"6f040abf-425e-4d21-9015-489f3dd38ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"
Nov 03 20:27:21.113274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:21.113246    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrr5\" (UniqueName: \"kubernetes.io/projected/6f040abf-425e-4d21-9015-489f3dd38ec5-kube-api-access-vzrr5\") pod \"maas-api-revoke-sub-abad4031-xmsnl\" (UID: \"6f040abf-425e-4d21-9015-489f3dd38ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"
Nov 03 20:27:21.254590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:21.254553    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"
Nov 03 20:27:21.515861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:21.515826    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:21.621556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:21.621526    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"]
Nov 03 20:27:21.922543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:21.922507    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl" event={"ID":"6f040abf-425e-4d21-9015-489f3dd38ec5","Type":"ContainerStarted","Data":"76cd997ae03f3b6076b73846213364dc418a356d49647411a9672ee38bd877a7"}
Nov 03 20:27:22.249958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:22.249922    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:22.934236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:22.934139    5659 generic.go:356] "Generic (PLEG): container finished" podID="6f040abf-425e-4d21-9015-489f3dd38ec5" containerID="ec913edaee931bc8bfb693e01ea37fdbbb7a74b4b07a7151d7293d4fe2c1f44e" exitCode=22
Nov 03 20:27:22.934415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:22.934278    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl" event={"ID":"6f040abf-425e-4d21-9015-489f3dd38ec5","Type":"ContainerDied","Data":"ec913edaee931bc8bfb693e01ea37fdbbb7a74b4b07a7151d7293d4fe2c1f44e"}
Nov 03 20:27:22.934728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:22.934706    5659 scope.go:122] "RemoveContainer" containerID="ec913edaee931bc8bfb693e01ea37fdbbb7a74b4b07a7151d7293d4fe2c1f44e"
Nov 03 20:27:22.939794 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:22.939736    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:23.636070 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:23.635919    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:23.948094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:23.948007    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl" event={"ID":"6f040abf-425e-4d21-9015-489f3dd38ec5","Type":"ContainerStarted","Data":"e05c0ec017b82fa6d04c149623e86f3da430fb2f7e58404d0bc99e3bfa045458"}
Nov 03 20:27:23.972019 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:23.971964    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl" podStartSLOduration=3.971949194 podStartE2EDuration="3.971949194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:23.969164579 +0000 UTC m=+2996.124066999" watchObservedRunningTime="2026-11-03 20:27:23.971949194 +0000 UTC m=+2996.126851652"
Nov 03 20:27:24.020154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:24.020112    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:27:24.339229 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:24.339195    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:24.962429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:24.962391    5659 generic.go:356] "Generic (PLEG): container finished" podID="6f040abf-425e-4d21-9015-489f3dd38ec5" containerID="e05c0ec017b82fa6d04c149623e86f3da430fb2f7e58404d0bc99e3bfa045458" exitCode=22
Nov 03 20:27:24.962959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:24.962476    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl" event={"ID":"6f040abf-425e-4d21-9015-489f3dd38ec5","Type":"ContainerDied","Data":"e05c0ec017b82fa6d04c149623e86f3da430fb2f7e58404d0bc99e3bfa045458"}
Nov 03 20:27:24.962959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:24.962575    5659 scope.go:122] "RemoveContainer" containerID="ec913edaee931bc8bfb693e01ea37fdbbb7a74b4b07a7151d7293d4fe2c1f44e"
Nov 03 20:27:24.962959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:24.962773    5659 scope.go:122] "RemoveContainer" containerID="e05c0ec017b82fa6d04c149623e86f3da430fb2f7e58404d0bc99e3bfa045458"
Nov 03 20:27:24.963176 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:24.963150    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-xmsnl_odh-ai-gateway-infra(6f040abf-425e-4d21-9015-489f3dd38ec5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl" podUID="6f040abf-425e-4d21-9015-489f3dd38ec5"
Nov 03 20:27:25.038802 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:25.038768    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:25.727676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:25.727642    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:25.973720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:25.973690    5659 scope.go:122] "RemoveContainer" containerID="e05c0ec017b82fa6d04c149623e86f3da430fb2f7e58404d0bc99e3bfa045458"
Nov 03 20:27:25.974190 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:25.974167    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-xmsnl_odh-ai-gateway-infra(6f040abf-425e-4d21-9015-489f3dd38ec5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl" podUID="6f040abf-425e-4d21-9015-489f3dd38ec5"
Nov 03 20:27:26.378417 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:26.378374    5659 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:27:26.378672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:26.378444    5659 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:27:26.379098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:26.379068    5659 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"8dc51c52507b3abd5b29387b230d0ca0bd06daecb9ffa9ffbc3a1618b6402f66"} pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" containerMessage="Container maas-api failed liveness probe, will be restarted"
Nov 03 20:27:26.379098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:26.379089    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Nov 03 20:27:26.379275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:26.379124    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api" containerID="cri-o://8dc51c52507b3abd5b29387b230d0ca0bd06daecb9ffa9ffbc3a1618b6402f66" gracePeriod=30
Nov 03 20:27:26.469710 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:26.469673    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:26.983455 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:26.983419    5659 generic.go:356] "Generic (PLEG): container finished" podID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerID="8dc51c52507b3abd5b29387b230d0ca0bd06daecb9ffa9ffbc3a1618b6402f66" exitCode=2
Nov 03 20:27:26.983900 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:26.983502    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" event={"ID":"59b60f28-7800-45ba-9fa6-3964633c5df3","Type":"ContainerDied","Data":"8dc51c52507b3abd5b29387b230d0ca0bd06daecb9ffa9ffbc3a1618b6402f66"}
Nov 03 20:27:27.191844 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:27.191696    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:27.868519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:27.868403    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:27.993322 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:27.993290    5659 generic.go:356] "Generic (PLEG): container finished" podID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerID="bdb7c1df5b43a1a57bbf307b769c65022fe3206b9838db21c1f131a4d656868b" exitCode=1
Nov 03 20:27:27.993830 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:27.993374    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" event={"ID":"59b60f28-7800-45ba-9fa6-3964633c5df3","Type":"ContainerDied","Data":"bdb7c1df5b43a1a57bbf307b769c65022fe3206b9838db21c1f131a4d656868b"}
Nov 03 20:27:27.993830 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:27.993442    5659 scope.go:122] "RemoveContainer" containerID="8dc51c52507b3abd5b29387b230d0ca0bd06daecb9ffa9ffbc3a1618b6402f66"
Nov 03 20:27:27.993830 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:27.993735    5659 scope.go:122] "RemoveContainer" containerID="bdb7c1df5b43a1a57bbf307b769c65022fe3206b9838db21c1f131a4d656868b"
Nov 03 20:27:28.525742 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:28.525713    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:29.003266 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:29.003226    5659 generic.go:356] "Generic (PLEG): container finished" podID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerID="ad6022fbab06cff83c3f300ab50afe57b3bb3984096f52210d16c48c38ca13cf" exitCode=1
Nov 03 20:27:29.003793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:29.003324    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" event={"ID":"59b60f28-7800-45ba-9fa6-3964633c5df3","Type":"ContainerDied","Data":"ad6022fbab06cff83c3f300ab50afe57b3bb3984096f52210d16c48c38ca13cf"}
Nov 03 20:27:29.003793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:29.003398    5659 scope.go:122] "RemoveContainer" containerID="bdb7c1df5b43a1a57bbf307b769c65022fe3206b9838db21c1f131a4d656868b"
Nov 03 20:27:29.003793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:29.003674    5659 scope.go:122] "RemoveContainer" containerID="ad6022fbab06cff83c3f300ab50afe57b3bb3984096f52210d16c48c38ca13cf"
Nov 03 20:27:29.003983 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:29.003957    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr_odh-ai-gateway-infra(59b60f28-7800-45ba-9fa6-3964633c5df3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3"
Nov 03 20:27:29.262548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:29.262512    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:30.018503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:30.011717    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:30.046214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:30.046185    5659 scope.go:122] "RemoveContainer" containerID="ad6022fbab06cff83c3f300ab50afe57b3bb3984096f52210d16c48c38ca13cf"
Nov 03 20:27:30.046574 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:30.046550    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr_odh-ai-gateway-infra(59b60f28-7800-45ba-9fa6-3964633c5df3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3"
Nov 03 20:27:30.136562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:30.136532    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:30.628736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:30.628695    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:31.075996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:31.075953    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:31.381736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:31.381647    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:27:31.393263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:31.382945    5659 scope.go:122] "RemoveContainer" containerID="ad6022fbab06cff83c3f300ab50afe57b3bb3984096f52210d16c48c38ca13cf"
Nov 03 20:27:31.394331 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:31.394270    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr_odh-ai-gateway-infra(59b60f28-7800-45ba-9fa6-3964633c5df3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3"
Nov 03 20:27:31.447944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:31.444824    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:31.584672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:31.584611    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:32.173000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:32.172961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:32.864024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:32.863990    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:33.205696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:33.205656    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:33.339966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:33.339930    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:33.553611 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:33.553468    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:33.753895 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:33.753857    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:33.904917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:33.904827    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:34.376841 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:34.376807    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:35.141405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:35.141358    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:35.862034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:35.861997    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:35.976084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:35.976049    5659 scope.go:122] "RemoveContainer" containerID="e05c0ec017b82fa6d04c149623e86f3da430fb2f7e58404d0bc99e3bfa045458"
Nov 03 20:27:36.378512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:36.378468    5659 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:27:36.379372 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:36.379344    5659 scope.go:122] "RemoveContainer" containerID="ad6022fbab06cff83c3f300ab50afe57b3bb3984096f52210d16c48c38ca13cf"
Nov 03 20:27:36.379881 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:36.379851    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr_odh-ai-gateway-infra(59b60f28-7800-45ba-9fa6-3964633c5df3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3"
Nov 03 20:27:36.692566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:36.689303    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:37.127015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:37.126980    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl" event={"ID":"6f040abf-425e-4d21-9015-489f3dd38ec5","Type":"ContainerStarted","Data":"8ff3d61febd3e9f2981bc12b32346c62fa5dfc376529b07d3917aa10468c9a5a"}
Nov 03 20:27:37.337273 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:37.337199    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:38.138293 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:38.138255    5659 generic.go:356] "Generic (PLEG): container finished" podID="6f040abf-425e-4d21-9015-489f3dd38ec5" containerID="8ff3d61febd3e9f2981bc12b32346c62fa5dfc376529b07d3917aa10468c9a5a" exitCode=22
Nov 03 20:27:38.138847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:38.138330    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl" event={"ID":"6f040abf-425e-4d21-9015-489f3dd38ec5","Type":"ContainerDied","Data":"8ff3d61febd3e9f2981bc12b32346c62fa5dfc376529b07d3917aa10468c9a5a"}
Nov 03 20:27:38.138847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:38.138412    5659 scope.go:122] "RemoveContainer" containerID="e05c0ec017b82fa6d04c149623e86f3da430fb2f7e58404d0bc99e3bfa045458"
Nov 03 20:27:38.139059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:38.139033    5659 scope.go:122] "RemoveContainer" containerID="8ff3d61febd3e9f2981bc12b32346c62fa5dfc376529b07d3917aa10468c9a5a"
Nov 03 20:27:38.139953 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:38.139544    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-xmsnl_odh-ai-gateway-infra(6f040abf-425e-4d21-9015-489f3dd38ec5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl" podUID="6f040abf-425e-4d21-9015-489f3dd38ec5"
Nov 03 20:27:38.163212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:38.163179    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"]
Nov 03 20:27:38.182026 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:38.181976    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:38.981684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:38.981651    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:39.341800 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:39.341772    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"
Nov 03 20:27:39.425838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:39.425806    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-openshift-ca\") pod \"6f040abf-425e-4d21-9015-489f3dd38ec5\" (UID: \"6f040abf-425e-4d21-9015-489f3dd38ec5\") "
Nov 03 20:27:39.426130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:39.426113    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-xks-ca\") pod \"6f040abf-425e-4d21-9015-489f3dd38ec5\" (UID: \"6f040abf-425e-4d21-9015-489f3dd38ec5\") "
Nov 03 20:27:39.426202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:39.426142    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-openshift-ca" pod "6f040abf-425e-4d21-9015-489f3dd38ec5" (UID: "6f040abf-425e-4d21-9015-489f3dd38ec5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:39.426282 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:39.426263    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f040abf-425e-4d21-9015-489f3dd38ec5-kube-api-access-vzrr5\" (UniqueName: \"kubernetes.io/projected/6f040abf-425e-4d21-9015-489f3dd38ec5-kube-api-access-vzrr5\") pod \"6f040abf-425e-4d21-9015-489f3dd38ec5\" (UID: \"6f040abf-425e-4d21-9015-489f3dd38ec5\") "
Nov 03 20:27:39.427242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:39.427180    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:39.428447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:39.428426    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f040abf-425e-4d21-9015-489f3dd38ec5-kube-api-access-vzrr5" pod "6f040abf-425e-4d21-9015-489f3dd38ec5" (UID: "6f040abf-425e-4d21-9015-489f3dd38ec5"). InnerVolumeSpecName "kube-api-access-vzrr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:39.428715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:39.428695    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-xks-ca" pod "6f040abf-425e-4d21-9015-489f3dd38ec5" (UID: "6f040abf-425e-4d21-9015-489f3dd38ec5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:39.528889 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:39.528855    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f040abf-425e-4d21-9015-489f3dd38ec5-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:39.528889 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:39.528885    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrr5\" (UniqueName: \"kubernetes.io/projected/6f040abf-425e-4d21-9015-489f3dd38ec5-kube-api-access-vzrr5\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:39.840149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:39.840045    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:39.962694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:39.962663    5659 scope.go:122] "RemoveContainer" containerID="ad6022fbab06cff83c3f300ab50afe57b3bb3984096f52210d16c48c38ca13cf"
Nov 03 20:27:40.166196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.166086    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl" event={"ID":"6f040abf-425e-4d21-9015-489f3dd38ec5","Type":"ContainerDied","Data":"76cd997ae03f3b6076b73846213364dc418a356d49647411a9672ee38bd877a7"}
Nov 03 20:27:40.166521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.166448    5659 scope.go:122] "RemoveContainer" containerID="8ff3d61febd3e9f2981bc12b32346c62fa5dfc376529b07d3917aa10468c9a5a"
Nov 03 20:27:40.166646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.166087    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"
Nov 03 20:27:40.190198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.190165    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"]
Nov 03 20:27:40.194736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.194712    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-xmsnl"]
Nov 03 20:27:40.527588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.527545    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:40.884806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.884770    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"]
Nov 03 20:27:40.885660 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.885644    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6f040abf-425e-4d21-9015-489f3dd38ec5" containerName="revoke-keys"
Nov 03 20:27:40.885729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.885661    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f040abf-425e-4d21-9015-489f3dd38ec5" containerName="revoke-keys"
Nov 03 20:27:40.885729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.885678    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6f040abf-425e-4d21-9015-489f3dd38ec5" containerName="revoke-keys"
Nov 03 20:27:40.885729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.885684    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f040abf-425e-4d21-9015-489f3dd38ec5" containerName="revoke-keys"
Nov 03 20:27:40.885729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.885696    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6f040abf-425e-4d21-9015-489f3dd38ec5" containerName="revoke-keys"
Nov 03 20:27:40.885729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.885702    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f040abf-425e-4d21-9015-489f3dd38ec5" containerName="revoke-keys"
Nov 03 20:27:40.885929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.885836    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f040abf-425e-4d21-9015-489f3dd38ec5" containerName="revoke-keys"
Nov 03 20:27:40.885929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.885845    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f040abf-425e-4d21-9015-489f3dd38ec5" containerName="revoke-keys"
Nov 03 20:27:40.889764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.889744    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"
Nov 03 20:27:40.894439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.893587    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:27:40.894738 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.894718    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\""
Nov 03 20:27:40.898371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.898349    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"]
Nov 03 20:27:40.969040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.968943    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g56fn\" (UniqueName: \"kubernetes.io/projected/c0273aaa-965c-4bc5-b34e-32e792396ead-kube-api-access-g56fn\") pod \"maas-api-revoke-sub-abad4031-fc8nk\" (UID: \"c0273aaa-965c-4bc5-b34e-32e792396ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"
Nov 03 20:27:40.969207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.969096    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-fc8nk\" (UID: \"c0273aaa-965c-4bc5-b34e-32e792396ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"
Nov 03 20:27:40.969207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:40.969179    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-fc8nk\" (UID: \"c0273aaa-965c-4bc5-b34e-32e792396ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"
Nov 03 20:27:41.071014 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.070963    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g56fn\" (UniqueName: \"kubernetes.io/projected/c0273aaa-965c-4bc5-b34e-32e792396ead-kube-api-access-g56fn\") pod \"maas-api-revoke-sub-abad4031-fc8nk\" (UID: \"c0273aaa-965c-4bc5-b34e-32e792396ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"
Nov 03 20:27:41.071191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.071095    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-fc8nk\" (UID: \"c0273aaa-965c-4bc5-b34e-32e792396ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"
Nov 03 20:27:41.071191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.071179    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-fc8nk\" (UID: \"c0273aaa-965c-4bc5-b34e-32e792396ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"
Nov 03 20:27:41.072092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.072063    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-fc8nk\" (UID: \"c0273aaa-965c-4bc5-b34e-32e792396ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"
Nov 03 20:27:41.073904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.073879    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-fc8nk\" (UID: \"c0273aaa-965c-4bc5-b34e-32e792396ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"
Nov 03 20:27:41.083745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.083716    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g56fn\" (UniqueName: \"kubernetes.io/projected/c0273aaa-965c-4bc5-b34e-32e792396ead-kube-api-access-g56fn\") pod \"maas-api-revoke-sub-abad4031-fc8nk\" (UID: \"c0273aaa-965c-4bc5-b34e-32e792396ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"
Nov 03 20:27:41.180554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.180515    5659 generic.go:356] "Generic (PLEG): container finished" podID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerID="b04d8748524fc101403d7ad1ea69ccd25a9295d38c5a813ac130ae06a364fc82" exitCode=1
Nov 03 20:27:41.180554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.180540    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" event={"ID":"59b60f28-7800-45ba-9fa6-3964633c5df3","Type":"ContainerDied","Data":"b04d8748524fc101403d7ad1ea69ccd25a9295d38c5a813ac130ae06a364fc82"}
Nov 03 20:27:41.180801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.180601    5659 scope.go:122] "RemoveContainer" containerID="ad6022fbab06cff83c3f300ab50afe57b3bb3984096f52210d16c48c38ca13cf"
Nov 03 20:27:41.181066 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.181048    5659 scope.go:122] "RemoveContainer" containerID="b04d8748524fc101403d7ad1ea69ccd25a9295d38c5a813ac130ae06a364fc82"
Nov 03 20:27:41.181558 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:41.181426    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr_odh-ai-gateway-infra(59b60f28-7800-45ba-9fa6-3964633c5df3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3"
Nov 03 20:27:41.214518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.214477    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"
Nov 03 20:27:41.345252 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.345218    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:41.418609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.395599    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:27:41.518003 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.517781    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"]
Nov 03 20:27:41.518161 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:27:41.518010    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0273aaa_965c_4bc5_b34e_32e792396ead.slice/crio-deafa73cb2e7c8f578d52e850a0aec2040d7793c268e1d6460e131e649374eed WatchSource:0}: Error finding container deafa73cb2e7c8f578d52e850a0aec2040d7793c268e1d6460e131e649374eed: Status 404 returned error can't find the container with id deafa73cb2e7c8f578d52e850a0aec2040d7793c268e1d6460e131e649374eed
Nov 03 20:27:41.971775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:41.971741    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f040abf-425e-4d21-9015-489f3dd38ec5" path="/var/lib/kubelet/pods/6f040abf-425e-4d21-9015-489f3dd38ec5/volumes"
Nov 03 20:27:42.068334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:42.068303    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:42.194529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:42.194418    5659 scope.go:122] "RemoveContainer" containerID="b04d8748524fc101403d7ad1ea69ccd25a9295d38c5a813ac130ae06a364fc82"
Nov 03 20:27:42.194905 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:42.194876    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr_odh-ai-gateway-infra(59b60f28-7800-45ba-9fa6-3964633c5df3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3"
Nov 03 20:27:42.195833 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:42.195805    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk" event={"ID":"c0273aaa-965c-4bc5-b34e-32e792396ead","Type":"ContainerStarted","Data":"c1ab0e7b6c67594fc4b427b0c36639834aa1f1ea7829df5c04c3a7758523560a"}
Nov 03 20:27:42.195943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:42.195857    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk" event={"ID":"c0273aaa-965c-4bc5-b34e-32e792396ead","Type":"ContainerStarted","Data":"deafa73cb2e7c8f578d52e850a0aec2040d7793c268e1d6460e131e649374eed"}
Nov 03 20:27:42.240824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:42.240764    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk" podStartSLOduration=2.240746608 podStartE2EDuration="2.240746608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:42.237148769 +0000 UTC m=+3014.392051191" watchObservedRunningTime="2026-11-03 20:27:42.240746608 +0000 UTC m=+3014.395649028"
Nov 03 20:27:42.836088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:42.836007    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:43.206161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:43.206127    5659 generic.go:356] "Generic (PLEG): container finished" podID="c0273aaa-965c-4bc5-b34e-32e792396ead" containerID="c1ab0e7b6c67594fc4b427b0c36639834aa1f1ea7829df5c04c3a7758523560a" exitCode=22
Nov 03 20:27:43.206582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:43.206214    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk" event={"ID":"c0273aaa-965c-4bc5-b34e-32e792396ead","Type":"ContainerDied","Data":"c1ab0e7b6c67594fc4b427b0c36639834aa1f1ea7829df5c04c3a7758523560a"}
Nov 03 20:27:43.206582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:43.206546    5659 scope.go:122] "RemoveContainer" containerID="c1ab0e7b6c67594fc4b427b0c36639834aa1f1ea7829df5c04c3a7758523560a"
Nov 03 20:27:43.637202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:43.637167    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:44.216466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:44.216429    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk" event={"ID":"c0273aaa-965c-4bc5-b34e-32e792396ead","Type":"ContainerStarted","Data":"8a23c06cf3f32ebef0bfe645c2e6b8c6f88ac6b15b9d94bd6dd29487e019ae2f"}
Nov 03 20:27:44.325172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:44.325135    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:45.107751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:45.107712    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:45.223961 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:45.223928    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:27:45.228057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:45.228032    5659 generic.go:356] "Generic (PLEG): container finished" podID="c0273aaa-965c-4bc5-b34e-32e792396ead" containerID="8a23c06cf3f32ebef0bfe645c2e6b8c6f88ac6b15b9d94bd6dd29487e019ae2f" exitCode=22
Nov 03 20:27:45.228215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:45.228182    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk" event={"ID":"c0273aaa-965c-4bc5-b34e-32e792396ead","Type":"ContainerDied","Data":"8a23c06cf3f32ebef0bfe645c2e6b8c6f88ac6b15b9d94bd6dd29487e019ae2f"}
Nov 03 20:27:45.228298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:45.228248    5659 scope.go:122] "RemoveContainer" containerID="c1ab0e7b6c67594fc4b427b0c36639834aa1f1ea7829df5c04c3a7758523560a"
Nov 03 20:27:45.228932 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:45.228913    5659 scope.go:122] "RemoveContainer" containerID="8a23c06cf3f32ebef0bfe645c2e6b8c6f88ac6b15b9d94bd6dd29487e019ae2f"
Nov 03 20:27:45.229538 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:45.229509    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-fc8nk_odh-ai-gateway-infra(c0273aaa-965c-4bc5-b34e-32e792396ead)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk" podUID="c0273aaa-965c-4bc5-b34e-32e792396ead"
Nov 03 20:27:45.734420 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:45.734092    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:46.240681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:46.240646    5659 scope.go:122] "RemoveContainer" containerID="8a23c06cf3f32ebef0bfe645c2e6b8c6f88ac6b15b9d94bd6dd29487e019ae2f"
Nov 03 20:27:46.241094 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:46.241045    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-fc8nk_odh-ai-gateway-infra(c0273aaa-965c-4bc5-b34e-32e792396ead)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk" podUID="c0273aaa-965c-4bc5-b34e-32e792396ead"
Nov 03 20:27:46.378589 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:46.378553    5659 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:27:46.379100 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:46.379084    5659 scope.go:122] "RemoveContainer" containerID="b04d8748524fc101403d7ad1ea69ccd25a9295d38c5a813ac130ae06a364fc82"
Nov 03 20:27:46.379510 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:46.379472    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr_odh-ai-gateway-infra(59b60f28-7800-45ba-9fa6-3964633c5df3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3"
Nov 03 20:27:46.454186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:46.454154    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:47.195262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:47.195067    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:47.899072 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:47.899015    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:48.640498 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:48.640458    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:49.468504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:49.467804    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:50.397505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:50.394972    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:51.257688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.251065    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:51.369550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.369518    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"]
Nov 03 20:27:51.370448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.370433    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f040abf-425e-4d21-9015-489f3dd38ec5" containerName="revoke-keys"
Nov 03 20:27:51.376049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.376021    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"
Nov 03 20:27:51.385256 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.385229    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"]
Nov 03 20:27:51.437522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.437451    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-wf2rb\" (UID: \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"
Nov 03 20:27:51.437704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.437640    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7x7s\" (UniqueName: \"kubernetes.io/projected/56aabd38-0e31-44ed-b489-f7eb2fc4e430-kube-api-access-n7x7s\") pod \"maas-api-revoke-sub-5bac6f1e-wf2rb\" (UID: \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"
Nov 03 20:27:51.438046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.438024    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-wf2rb\" (UID: \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"
Nov 03 20:27:51.539587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.539466    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-wf2rb\" (UID: \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"
Nov 03 20:27:51.539771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.539609    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-wf2rb\" (UID: \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"
Nov 03 20:27:51.539771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.539674    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7x7s\" (UniqueName: \"kubernetes.io/projected/56aabd38-0e31-44ed-b489-f7eb2fc4e430-kube-api-access-n7x7s\") pod \"maas-api-revoke-sub-5bac6f1e-wf2rb\" (UID: \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"
Nov 03 20:27:51.540604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.540577    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-wf2rb\" (UID: \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"
Nov 03 20:27:51.542414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.542387    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-wf2rb\" (UID: \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"
Nov 03 20:27:51.552329 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.552308    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7x7s\" (UniqueName: \"kubernetes.io/projected/56aabd38-0e31-44ed-b489-f7eb2fc4e430-kube-api-access-n7x7s\") pod \"maas-api-revoke-sub-5bac6f1e-wf2rb\" (UID: \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"
Nov 03 20:27:51.705066 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.705029    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"
Nov 03 20:27:51.993232 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:27:51.993202    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56aabd38_0e31_44ed_b489_f7eb2fc4e430.slice/crio-00a9bb0aa373cd4b04fdc6d8aa910b336d05bdd7573b8eb48502048a406afc8b WatchSource:0}: Error finding container 00a9bb0aa373cd4b04fdc6d8aa910b336d05bdd7573b8eb48502048a406afc8b: Status 404 returned error can't find the container with id 00a9bb0aa373cd4b04fdc6d8aa910b336d05bdd7573b8eb48502048a406afc8b
Nov 03 20:27:51.994184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:51.994082    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"]
Nov 03 20:27:52.039637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:52.039611    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:52.305356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:52.305317    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"]
Nov 03 20:27:52.306215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:52.306182    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy" containerID="cri-o://1cdef80f2f136980bda4e3d8a25fc85e80c5934887110c2bd9abe9fda27fd3b4" gracePeriod=30
Nov 03 20:27:52.367694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:52.367648    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb" event={"ID":"56aabd38-0e31-44ed-b489-f7eb2fc4e430","Type":"ContainerStarted","Data":"00a9bb0aa373cd4b04fdc6d8aa910b336d05bdd7573b8eb48502048a406afc8b"}
Nov 03 20:27:52.378204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:52.378176    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:52.717503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:52.715784    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:52.959575 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:52.952267    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:53.377159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:53.377074    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb" event={"ID":"56aabd38-0e31-44ed-b489-f7eb2fc4e430","Type":"ContainerStarted","Data":"fec05504562479e486913a4fb89e05496596f4880da3642a2ee728fee5ce8fd4"}
Nov 03 20:27:53.402316 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:53.402260    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb" podStartSLOduration=2.402246288 podStartE2EDuration="2.402246288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:53.397177971 +0000 UTC m=+3025.552080390" watchObservedRunningTime="2026-11-03 20:27:53.402246288 +0000 UTC m=+3025.557148707"
Nov 03 20:27:53.774517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:53.769086    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:54.399081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:54.399049    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:54.957308 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:54.957273    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:55.089203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:55.089171    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:55.242849 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:55.242738    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:56.067015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:56.064345    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:56.813247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:56.813213    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:57.567580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.567541    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:57.721747 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.721723    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:27:57.762967 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.762920    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"]
Nov 03 20:27:57.764289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.764265    5659 cpu_manager.go:414] "containerMap: removing container" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy"
Nov 03 20:27:57.764357 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.764293    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy"
Nov 03 20:27:57.764357 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.764344    5659 cpu_manager.go:414] "containerMap: removing container" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy"
Nov 03 20:27:57.764357 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.764353    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy"
Nov 03 20:27:57.764561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.764546    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy"
Nov 03 20:27:57.773310 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.769189    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.786729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.786698    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"]
Nov 03 20:27:57.814081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.814039    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.814281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.814261    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb414b62-f84b-4da8-9891-b85f75868dd0-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.814431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.814413    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.814565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.814462    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.814565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.814500    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.814665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.814635    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.814712 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.814687    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlsjs\" (UniqueName: \"kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-kube-api-access-jlsjs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.814777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.814760    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.814828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.814809    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.915580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.915544    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-socket\") pod \"4d0a991c-5faa-416f-be22-d7b258a358e1\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") "
Nov 03 20:27:57.915777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.915633    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-kube-api-access-df7dq\" (UniqueName: \"kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-kube-api-access-df7dq\") pod \"4d0a991c-5faa-416f-be22-d7b258a358e1\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") "
Nov 03 20:27:57.915777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.915668    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-credential-socket\") pod \"4d0a991c-5faa-416f-be22-d7b258a358e1\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") "
Nov 03 20:27:57.915777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.915698    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-envoy\") pod \"4d0a991c-5faa-416f-be22-d7b258a358e1\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") "
Nov 03 20:27:57.915777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.915728    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-certs\") pod \"4d0a991c-5faa-416f-be22-d7b258a358e1\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") "
Nov 03 20:27:57.915777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.915754    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-data\") pod \"4d0a991c-5faa-416f-be22-d7b258a358e1\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") "
Nov 03 20:27:57.916084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.915861    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-socket" pod "4d0a991c-5faa-416f-be22-d7b258a358e1" (UID: "4d0a991c-5faa-416f-be22-d7b258a358e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:57.916084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.915897    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0a991c-5faa-416f-be22-d7b258a358e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0a991c-5faa-416f-be22-d7b258a358e1-istiod-ca-cert\") pod \"4d0a991c-5faa-416f-be22-d7b258a358e1\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") "
Nov 03 20:27:57.916084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.916011    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-token\" (UniqueName: \"kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-token\") pod \"4d0a991c-5faa-416f-be22-d7b258a358e1\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") "
Nov 03 20:27:57.916084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.916030    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-certs" pod "4d0a991c-5faa-416f-be22-d7b258a358e1" (UID: "4d0a991c-5faa-416f-be22-d7b258a358e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:57.916084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.916045    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-credential-socket" pod "4d0a991c-5faa-416f-be22-d7b258a358e1" (UID: "4d0a991c-5faa-416f-be22-d7b258a358e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:57.916084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.916070    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-data" pod "4d0a991c-5faa-416f-be22-d7b258a358e1" (UID: "4d0a991c-5faa-416f-be22-d7b258a358e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:57.916415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.916247    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-podinfo\") pod \"4d0a991c-5faa-416f-be22-d7b258a358e1\" (UID: \"4d0a991c-5faa-416f-be22-d7b258a358e1\") "
Nov 03 20:27:57.916415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.916259    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0a991c-5faa-416f-be22-d7b258a358e1-istiod-ca-cert" pod "4d0a991c-5faa-416f-be22-d7b258a358e1" (UID: "4d0a991c-5faa-416f-be22-d7b258a358e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:57.916697 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.916667    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlsjs\" (UniqueName: \"kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-kube-api-access-jlsjs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.916788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.916711    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.916788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.916756    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.916918 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.916861    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.917160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.917030    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb414b62-f84b-4da8-9891-b85f75868dd0-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.917237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.917174    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.917237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.917227    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.917382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.917253    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.917382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.917308    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.917382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.917350    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.917581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.917394    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.917798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.917771    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.918345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.918322    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.919340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.918373    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:57.919340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.918398    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:57.919340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.918415    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:57.919340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.918432    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:57.919340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.918446    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0a991c-5faa-416f-be22-d7b258a358e1-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:57.919340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.918510    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb414b62-f84b-4da8-9891-b85f75868dd0-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.919340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.918633    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-kube-api-access-df7dq" pod "4d0a991c-5faa-416f-be22-d7b258a358e1" (UID: "4d0a991c-5faa-416f-be22-d7b258a358e1"). InnerVolumeSpecName "kube-api-access-df7dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:57.919340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.918710    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-envoy" pod "4d0a991c-5faa-416f-be22-d7b258a358e1" (UID: "4d0a991c-5faa-416f-be22-d7b258a358e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:57.919340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.918842    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-token" pod "4d0a991c-5faa-416f-be22-d7b258a358e1" (UID: "4d0a991c-5faa-416f-be22-d7b258a358e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:57.919340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.918850    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-podinfo" pod "4d0a991c-5faa-416f-be22-d7b258a358e1" (UID: "4d0a991c-5faa-416f-be22-d7b258a358e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:57.919924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.919901    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.920565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.920547    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.929868 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.929839    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.929953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.929914    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlsjs\" (UniqueName: \"kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-kube-api-access-jlsjs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:57.983170 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:57.983138    5659 scope.go:122] "RemoveContainer" containerID="8a23c06cf3f32ebef0bfe645c2e6b8c6f88ac6b15b9d94bd6dd29487e019ae2f"
Nov 03 20:27:58.020421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.020394    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df7dq\" (UniqueName: \"kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-kube-api-access-df7dq\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:58.020523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.020426    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:58.020523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.020441    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:58.020523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.020454    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0a991c-5faa-416f-be22-d7b258a358e1-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:58.091828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.091793    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:27:58.251475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.251362    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:58.425256 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.424976    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"]
Nov 03 20:27:58.440541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.431733    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:27:58.440541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.437689    5659 generic.go:356] "Generic (PLEG): container finished" podID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerID="1cdef80f2f136980bda4e3d8a25fc85e80c5934887110c2bd9abe9fda27fd3b4" exitCode=0
Nov 03 20:27:58.440541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.437803    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" event={"ID":"4d0a991c-5faa-416f-be22-d7b258a358e1","Type":"ContainerDied","Data":"1cdef80f2f136980bda4e3d8a25fc85e80c5934887110c2bd9abe9fda27fd3b4"}
Nov 03 20:27:58.440541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.437861    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk" event={"ID":"4d0a991c-5faa-416f-be22-d7b258a358e1","Type":"ContainerDied","Data":"a196bb131059ccb0cba2e133af3913ca1346b7912c123fb6ffd2e66cbab307e1"}
Nov 03 20:27:58.440541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.437897    5659 scope.go:122] "RemoveContainer" containerID="1cdef80f2f136980bda4e3d8a25fc85e80c5934887110c2bd9abe9fda27fd3b4"
Nov 03 20:27:58.440541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.438094    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"
Nov 03 20:27:58.440541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.438667    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:27:58.440541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.438731    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:27:58.464621 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.463878    5659 scope.go:122] "RemoveContainer" containerID="d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de"
Nov 03 20:27:58.473780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.473751    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"]
Nov 03 20:27:58.489055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.489030    5659 scope.go:122] "RemoveContainer" containerID="1cdef80f2f136980bda4e3d8a25fc85e80c5934887110c2bd9abe9fda27fd3b4"
Nov 03 20:27:58.489474 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:58.489446    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cdef80f2f136980bda4e3d8a25fc85e80c5934887110c2bd9abe9fda27fd3b4\": container with ID starting with 1cdef80f2f136980bda4e3d8a25fc85e80c5934887110c2bd9abe9fda27fd3b4 not found: ID does not exist" containerID="1cdef80f2f136980bda4e3d8a25fc85e80c5934887110c2bd9abe9fda27fd3b4"
Nov 03 20:27:58.489607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.489527    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cdef80f2f136980bda4e3d8a25fc85e80c5934887110c2bd9abe9fda27fd3b4"} err="failed to get container status \"1cdef80f2f136980bda4e3d8a25fc85e80c5934887110c2bd9abe9fda27fd3b4\": rpc error: code = NotFound desc = could not find container \"1cdef80f2f136980bda4e3d8a25fc85e80c5934887110c2bd9abe9fda27fd3b4\": container with ID starting with 1cdef80f2f136980bda4e3d8a25fc85e80c5934887110c2bd9abe9fda27fd3b4 not found: ID does not exist"
Nov 03 20:27:58.489607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.489554    5659 scope.go:122] "RemoveContainer" containerID="d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de"
Nov 03 20:27:58.489990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.489968    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-28b78c-openshift-default-856b7f6fdc-kdmsk"]
Nov 03 20:27:58.490069 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:58.490002    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de\": container with ID starting with d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de not found: ID does not exist" containerID="d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de"
Nov 03 20:27:58.490069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.490031    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de"} err="failed to get container status \"d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de\": rpc error: code = NotFound desc = could not find container \"d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de\": container with ID starting with d74e09103bdc1ebc9529a103595ea6a9f0a31237a80899e2eb187c7cf44425de not found: ID does not exist"
Nov 03 20:27:58.932876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:58.932005    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:59.474319 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:59.474273    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk" event={"ID":"eb414b62-f84b-4da8-9891-b85f75868dd0","Type":"ContainerStarted","Data":"2af7ed40bd868991cc818d3c07409189994e01ea4556d31ecc0be76fc17e32a4"}
Nov 03 20:27:59.474585 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:59.474347    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk" event={"ID":"eb414b62-f84b-4da8-9891-b85f75868dd0","Type":"ContainerStarted","Data":"9250d8aa2d30caa19136e9e7d7efc70358e76c77a16296ab5674f977f7ab042f"}
Nov 03 20:27:59.478150 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:59.478116    5659 generic.go:356] "Generic (PLEG): container finished" podID="c0273aaa-965c-4bc5-b34e-32e792396ead" containerID="78aa183462b3216449252467f8147632bedff15fec65b68060225cddec8fed1e" exitCode=22
Nov 03 20:27:59.478310 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:59.478172    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk" event={"ID":"c0273aaa-965c-4bc5-b34e-32e792396ead","Type":"ContainerDied","Data":"78aa183462b3216449252467f8147632bedff15fec65b68060225cddec8fed1e"}
Nov 03 20:27:59.478310 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:59.478239    5659 scope.go:122] "RemoveContainer" containerID="8a23c06cf3f32ebef0bfe645c2e6b8c6f88ac6b15b9d94bd6dd29487e019ae2f"
Nov 03 20:27:59.478701 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:59.478684    5659 scope.go:122] "RemoveContainer" containerID="78aa183462b3216449252467f8147632bedff15fec65b68060225cddec8fed1e"
Nov 03 20:27:59.479119 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:27:59.479093    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-fc8nk_odh-ai-gateway-infra(c0273aaa-965c-4bc5-b34e-32e792396ead)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk" podUID="c0273aaa-965c-4bc5-b34e-32e792396ead"
Nov 03 20:27:59.503526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:59.503455    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk" podStartSLOduration=235.503442238 podStartE2EDuration="3m55.503442238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:59.500181483 +0000 UTC m=+3031.655083904" watchObservedRunningTime="2026-11-03 20:27:59.503442238 +0000 UTC m=+3031.658344730"
Nov 03 20:27:59.664505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:59.664119    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:59.991305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:27:59.991268    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" path="/var/lib/kubelet/pods/4d0a991c-5faa-416f-be22-d7b258a358e1/volumes"
Nov 03 20:28:00.092951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:00.092912    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:28:00.095085 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:00.095055    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 20:28:00.095219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:00.095110    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk" podUID="eb414b62-f84b-4da8-9891-b85f75868dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 20:28:00.461472 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:00.456978    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:00.601632 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:00.601589    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"]
Nov 03 20:28:00.893095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:00.893068    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"
Nov 03 20:28:00.993652 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:00.993614    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0273aaa-965c-4bc5-b34e-32e792396ead-kube-api-access-g56fn\" (UniqueName: \"kubernetes.io/projected/c0273aaa-965c-4bc5-b34e-32e792396ead-kube-api-access-g56fn\") pod \"c0273aaa-965c-4bc5-b34e-32e792396ead\" (UID: \"c0273aaa-965c-4bc5-b34e-32e792396ead\") "
Nov 03 20:28:00.994143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:00.993675    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-xks-ca\") pod \"c0273aaa-965c-4bc5-b34e-32e792396ead\" (UID: \"c0273aaa-965c-4bc5-b34e-32e792396ead\") "
Nov 03 20:28:00.994143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:00.993849    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-openshift-ca\") pod \"c0273aaa-965c-4bc5-b34e-32e792396ead\" (UID: \"c0273aaa-965c-4bc5-b34e-32e792396ead\") "
Nov 03 20:28:00.995029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:00.994996    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-openshift-ca" pod "c0273aaa-965c-4bc5-b34e-32e792396ead" (UID: "c0273aaa-965c-4bc5-b34e-32e792396ead"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:00.996436 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:00.996410    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-xks-ca" pod "c0273aaa-965c-4bc5-b34e-32e792396ead" (UID: "c0273aaa-965c-4bc5-b34e-32e792396ead"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:00.996554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:00.996524    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0273aaa-965c-4bc5-b34e-32e792396ead-kube-api-access-g56fn" pod "c0273aaa-965c-4bc5-b34e-32e792396ead" (UID: "c0273aaa-965c-4bc5-b34e-32e792396ead"). InnerVolumeSpecName "kube-api-access-g56fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:01.092692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:01.092595    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 20:28:01.092692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:01.092655    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk" podUID="eb414b62-f84b-4da8-9891-b85f75868dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 20:28:01.096960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:01.096931    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g56fn\" (UniqueName: \"kubernetes.io/projected/c0273aaa-965c-4bc5-b34e-32e792396ead-kube-api-access-g56fn\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:01.096960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:01.096963    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:01.097171 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:01.096978    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0273aaa-965c-4bc5-b34e-32e792396ead-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:01.267509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:01.267148    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:01.595626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:01.595594    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"
Nov 03 20:28:01.595849 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:01.595626    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk" event={"ID":"c0273aaa-965c-4bc5-b34e-32e792396ead","Type":"ContainerDied","Data":"deafa73cb2e7c8f578d52e850a0aec2040d7793c268e1d6460e131e649374eed"}
Nov 03 20:28:01.595849 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:01.595719    5659 scope.go:122] "RemoveContainer" containerID="78aa183462b3216449252467f8147632bedff15fec65b68060225cddec8fed1e"
Nov 03 20:28:01.626052 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:01.626020    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"]
Nov 03 20:28:01.630306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:01.630285    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fc8nk"]
Nov 03 20:28:01.956538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:01.956506    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:01.968403 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:01.968368    5659 scope.go:122] "RemoveContainer" containerID="b04d8748524fc101403d7ad1ea69ccd25a9295d38c5a813ac130ae06a364fc82"
Nov 03 20:28:02.024516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:02.024466    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0273aaa-965c-4bc5-b34e-32e792396ead" path="/var/lib/kubelet/pods/c0273aaa-965c-4bc5-b34e-32e792396ead/volumes"
Nov 03 20:28:02.093278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:02.093228    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 20:28:02.093534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:02.093294    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk" podUID="eb414b62-f84b-4da8-9891-b85f75868dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 20:28:02.606238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:02.606202    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" event={"ID":"59b60f28-7800-45ba-9fa6-3964633c5df3","Type":"ContainerStarted","Data":"0932c977f449700e2547ce96a785cf5ecd868a615d9a0157d839cb760c780d73"}
Nov 03 20:28:02.606457 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:02.606440    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:28:02.629928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:02.629879    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:03.092712 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:03.092623    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 20:28:03.092712 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:03.092678    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk" podUID="eb414b62-f84b-4da8-9891-b85f75868dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 20:28:03.362562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:03.362047    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:03.617417 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:03.617333    5659 generic.go:356] "Generic (PLEG): container finished" podID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerID="0932c977f449700e2547ce96a785cf5ecd868a615d9a0157d839cb760c780d73" exitCode=1
Nov 03 20:28:03.617620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:03.617413    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" event={"ID":"59b60f28-7800-45ba-9fa6-3964633c5df3","Type":"ContainerDied","Data":"0932c977f449700e2547ce96a785cf5ecd868a615d9a0157d839cb760c780d73"}
Nov 03 20:28:03.617620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:03.617475    5659 scope.go:122] "RemoveContainer" containerID="b04d8748524fc101403d7ad1ea69ccd25a9295d38c5a813ac130ae06a364fc82"
Nov 03 20:28:03.617782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:03.617766    5659 scope.go:122] "RemoveContainer" containerID="0932c977f449700e2547ce96a785cf5ecd868a615d9a0157d839cb760c780d73"
Nov 03 20:28:03.618142 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:03.618112    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr_odh-ai-gateway-infra(59b60f28-7800-45ba-9fa6-3964633c5df3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3"
Nov 03 20:28:04.051589 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:04.050129    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:04.117872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:04.108739    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 20:28:04.117872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:04.108804    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk" podUID="eb414b62-f84b-4da8-9891-b85f75868dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 20:28:04.627945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:04.627920    5659 scope.go:122] "RemoveContainer" containerID="0932c977f449700e2547ce96a785cf5ecd868a615d9a0157d839cb760c780d73"
Nov 03 20:28:04.628317 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:04.628296    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr_odh-ai-gateway-infra(59b60f28-7800-45ba-9fa6-3964633c5df3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3"
Nov 03 20:28:04.762346 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:04.762306    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:05.092533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:05.092466    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 20:28:05.092722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:05.092556    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk" podUID="eb414b62-f84b-4da8-9891-b85f75868dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 20:28:05.531413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:05.531373    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:06.098174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:06.098133    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:28:06.098558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:06.098535    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:28:06.099532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:06.099507    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:28:06.238157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:06.238123    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:06.379074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:06.378995    5659 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:28:06.379688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:06.379668    5659 scope.go:122] "RemoveContainer" containerID="0932c977f449700e2547ce96a785cf5ecd868a615d9a0157d839cb760c780d73"
Nov 03 20:28:06.380204 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:06.380161    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr_odh-ai-gateway-infra(59b60f28-7800-45ba-9fa6-3964633c5df3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3"
Nov 03 20:28:06.953925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:06.953877    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:07.629167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:07.629131    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:08.488572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:08.484164    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:09.425996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:09.425951    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:10.495938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:10.495660    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:11.109512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.108913    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"]
Nov 03 20:28:11.113438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.110153    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c0273aaa-965c-4bc5-b34e-32e792396ead" containerName="revoke-keys"
Nov 03 20:28:11.113438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.110177    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0273aaa-965c-4bc5-b34e-32e792396ead" containerName="revoke-keys"
Nov 03 20:28:11.113438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.110272    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c0273aaa-965c-4bc5-b34e-32e792396ead" containerName="revoke-keys"
Nov 03 20:28:11.113438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.110281    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0273aaa-965c-4bc5-b34e-32e792396ead" containerName="revoke-keys"
Nov 03 20:28:11.113438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.110476    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0273aaa-965c-4bc5-b34e-32e792396ead" containerName="revoke-keys"
Nov 03 20:28:11.113438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.110509    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0273aaa-965c-4bc5-b34e-32e792396ead" containerName="revoke-keys"
Nov 03 20:28:11.113438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.110532    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0a991c-5faa-416f-be22-d7b258a358e1" containerName="istio-proxy"
Nov 03 20:28:11.207586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.204713    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"]
Nov 03 20:28:11.207586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.204851    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"
Nov 03 20:28:11.289541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.289251    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:11.331234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.331202    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p4hz\" (UniqueName: \"kubernetes.io/projected/6e223b9a-8fcf-4870-8452-a001ff12f980-kube-api-access-8p4hz\") pod \"maas-api-revoke-sub-abad4031-fqdl5\" (UID: \"6e223b9a-8fcf-4870-8452-a001ff12f980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"
Nov 03 20:28:11.351564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.331530    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-fqdl5\" (UID: \"6e223b9a-8fcf-4870-8452-a001ff12f980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"
Nov 03 20:28:11.351564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.331612    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-fqdl5\" (UID: \"6e223b9a-8fcf-4870-8452-a001ff12f980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"
Nov 03 20:28:11.447211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.441959    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-fqdl5\" (UID: \"6e223b9a-8fcf-4870-8452-a001ff12f980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"
Nov 03 20:28:11.447211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.446226    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-fqdl5\" (UID: \"6e223b9a-8fcf-4870-8452-a001ff12f980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"
Nov 03 20:28:11.447211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.446421    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4hz\" (UniqueName: \"kubernetes.io/projected/6e223b9a-8fcf-4870-8452-a001ff12f980-kube-api-access-8p4hz\") pod \"maas-api-revoke-sub-abad4031-fqdl5\" (UID: \"6e223b9a-8fcf-4870-8452-a001ff12f980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"
Nov 03 20:28:11.447211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.447087    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-fqdl5\" (UID: \"6e223b9a-8fcf-4870-8452-a001ff12f980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"
Nov 03 20:28:11.470503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.463956    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-fqdl5\" (UID: \"6e223b9a-8fcf-4870-8452-a001ff12f980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"
Nov 03 20:28:11.494990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.483194    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4hz\" (UniqueName: \"kubernetes.io/projected/6e223b9a-8fcf-4870-8452-a001ff12f980-kube-api-access-8p4hz\") pod \"maas-api-revoke-sub-abad4031-fqdl5\" (UID: \"6e223b9a-8fcf-4870-8452-a001ff12f980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"
Nov 03 20:28:11.523974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.523945    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"
Nov 03 20:28:11.765531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.765496    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:11.932069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:11.905927    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"]
Nov 03 20:28:11.932069 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:28:11.906550    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e223b9a_8fcf_4870_8452_a001ff12f980.slice/crio-c0c016b3dcb490ae3d48ba0d9dcf1d6878d7f3dbf6afb7281f6b85673386c849 WatchSource:0}: Error finding container c0c016b3dcb490ae3d48ba0d9dcf1d6878d7f3dbf6afb7281f6b85673386c849: Status 404 returned error can't find the container with id c0c016b3dcb490ae3d48ba0d9dcf1d6878d7f3dbf6afb7281f6b85673386c849
Nov 03 20:28:12.782272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:12.773231    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:12.900198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:12.900156    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5" event={"ID":"6e223b9a-8fcf-4870-8452-a001ff12f980","Type":"ContainerStarted","Data":"5fb3cd7cb1190a806df94915b4c73f60d848ec5266e5086275bc207835ce21b8"}
Nov 03 20:28:12.900389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:12.900220    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5" event={"ID":"6e223b9a-8fcf-4870-8452-a001ff12f980","Type":"ContainerStarted","Data":"c0c016b3dcb490ae3d48ba0d9dcf1d6878d7f3dbf6afb7281f6b85673386c849"}
Nov 03 20:28:12.919949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:12.919889    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5" podStartSLOduration=1.919874734 podStartE2EDuration="1.919874734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:12.919106805 +0000 UTC m=+3045.074009243" watchObservedRunningTime="2026-11-03 20:28:12.919874734 +0000 UTC m=+3045.074777153"
Nov 03 20:28:13.427441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:13.427356    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:13.910071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:13.910034    5659 generic.go:356] "Generic (PLEG): container finished" podID="6e223b9a-8fcf-4870-8452-a001ff12f980" containerID="5fb3cd7cb1190a806df94915b4c73f60d848ec5266e5086275bc207835ce21b8" exitCode=22
Nov 03 20:28:13.910656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:13.910115    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5" event={"ID":"6e223b9a-8fcf-4870-8452-a001ff12f980","Type":"ContainerDied","Data":"5fb3cd7cb1190a806df94915b4c73f60d848ec5266e5086275bc207835ce21b8"}
Nov 03 20:28:13.910656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:13.910451    5659 scope.go:122] "RemoveContainer" containerID="5fb3cd7cb1190a806df94915b4c73f60d848ec5266e5086275bc207835ce21b8"
Nov 03 20:28:14.148673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:14.148643    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:14.826692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:14.826662    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:14.937981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:14.937514    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5" event={"ID":"6e223b9a-8fcf-4870-8452-a001ff12f980","Type":"ContainerStarted","Data":"5bf6061272a9f2de0b35ad0465987214a947def1da00bc44606b047d783a0812"}
Nov 03 20:28:15.221963 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:15.221915    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:28:15.632584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:15.630832    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:15.950634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:15.950535    5659 generic.go:356] "Generic (PLEG): container finished" podID="6e223b9a-8fcf-4870-8452-a001ff12f980" containerID="5bf6061272a9f2de0b35ad0465987214a947def1da00bc44606b047d783a0812" exitCode=22
Nov 03 20:28:15.950634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:15.950579    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5" event={"ID":"6e223b9a-8fcf-4870-8452-a001ff12f980","Type":"ContainerDied","Data":"5bf6061272a9f2de0b35ad0465987214a947def1da00bc44606b047d783a0812"}
Nov 03 20:28:15.951210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:15.950654    5659 scope.go:122] "RemoveContainer" containerID="5fb3cd7cb1190a806df94915b4c73f60d848ec5266e5086275bc207835ce21b8"
Nov 03 20:28:15.951210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:15.951022    5659 scope.go:122] "RemoveContainer" containerID="5bf6061272a9f2de0b35ad0465987214a947def1da00bc44606b047d783a0812"
Nov 03 20:28:15.951563 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:15.951532    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-fqdl5_odh-ai-gateway-infra(6e223b9a-8fcf-4870-8452-a001ff12f980)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5" podUID="6e223b9a-8fcf-4870-8452-a001ff12f980"
Nov 03 20:28:16.352098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:16.352060    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:16.963017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:16.962989    5659 scope.go:122] "RemoveContainer" containerID="5bf6061272a9f2de0b35ad0465987214a947def1da00bc44606b047d783a0812"
Nov 03 20:28:16.963437 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:16.963200    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-fqdl5_odh-ai-gateway-infra(6e223b9a-8fcf-4870-8452-a001ff12f980)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5" podUID="6e223b9a-8fcf-4870-8452-a001ff12f980"
Nov 03 20:28:17.094792 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:17.094756    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:17.852449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:17.852406    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:18.000314 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:18.000283    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:18.545808 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:18.545763    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:19.235015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:19.234958    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:20.067140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:20.067095    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:20.869848 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:20.869765    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:21.528773 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:21.528733    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:22.286505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:22.283201    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:22.700183 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:22.700070    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:22.843515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:22.836937    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:23.295708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:23.295667    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:23.439968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:23.439853    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:24.055754 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:24.055721    5659 generic.go:356] "Generic (PLEG): container finished" podID="56aabd38-0e31-44ed-b489-f7eb2fc4e430" containerID="fec05504562479e486913a4fb89e05496596f4880da3642a2ee728fee5ce8fd4" exitCode=28
Nov 03 20:28:24.055937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:24.055788    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb" event={"ID":"56aabd38-0e31-44ed-b489-f7eb2fc4e430","Type":"ContainerDied","Data":"fec05504562479e486913a4fb89e05496596f4880da3642a2ee728fee5ce8fd4"}
Nov 03 20:28:24.056233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:24.056219    5659 scope.go:122] "RemoveContainer" containerID="fec05504562479e486913a4fb89e05496596f4880da3642a2ee728fee5ce8fd4"
Nov 03 20:28:24.331640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:24.331547    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:25.008446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:25.008413    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:25.146504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:25.146436    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:25.160705 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:25.159775    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb" event={"ID":"56aabd38-0e31-44ed-b489-f7eb2fc4e430","Type":"ContainerStarted","Data":"333551521d0521be39f9cf1ff29055c135d0502a4be49032c035d0a3b27bdbb4"}
Nov 03 20:28:25.160705 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:25.160527    5659 scope.go:122] "RemoveContainer" containerID="333551521d0521be39f9cf1ff29055c135d0502a4be49032c035d0a3b27bdbb4"
Nov 03 20:28:25.162527 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:25.161959    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-wf2rb_odh-ai-gateway-infra(56aabd38-0e31-44ed-b489-f7eb2fc4e430)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb" podUID="56aabd38-0e31-44ed-b489-f7eb2fc4e430"
Nov 03 20:28:25.705279 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:25.705220    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:25.852732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:25.852695    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:26.171669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:26.171570    5659 generic.go:356] "Generic (PLEG): container finished" podID="56aabd38-0e31-44ed-b489-f7eb2fc4e430" containerID="333551521d0521be39f9cf1ff29055c135d0502a4be49032c035d0a3b27bdbb4" exitCode=22
Nov 03 20:28:26.171669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:26.171650    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb" event={"ID":"56aabd38-0e31-44ed-b489-f7eb2fc4e430","Type":"ContainerDied","Data":"333551521d0521be39f9cf1ff29055c135d0502a4be49032c035d0a3b27bdbb4"}
Nov 03 20:28:26.171910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:26.171726    5659 scope.go:122] "RemoveContainer" containerID="fec05504562479e486913a4fb89e05496596f4880da3642a2ee728fee5ce8fd4"
Nov 03 20:28:26.172269 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:26.172247    5659 scope.go:122] "RemoveContainer" containerID="333551521d0521be39f9cf1ff29055c135d0502a4be49032c035d0a3b27bdbb4"
Nov 03 20:28:26.173125 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:26.172718    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-wf2rb_odh-ai-gateway-infra(56aabd38-0e31-44ed-b489-f7eb2fc4e430)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb" podUID="56aabd38-0e31-44ed-b489-f7eb2fc4e430"
Nov 03 20:28:26.458447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:26.458406    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:27.170217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:27.170168    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:27.808725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:27.808691    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:27.969297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:27.969259    5659 scope.go:122] "RemoveContainer" containerID="5bf6061272a9f2de0b35ad0465987214a947def1da00bc44606b047d783a0812"
Nov 03 20:28:28.665588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:28.665505    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:29.259029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:29.258993    5659 generic.go:356] "Generic (PLEG): container finished" podID="6e223b9a-8fcf-4870-8452-a001ff12f980" containerID="3c7108aa875a803b226c93a15c517a317a1c30d631cfba0cd5496cf5168d620a" exitCode=22
Nov 03 20:28:29.259228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:29.259064    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5" event={"ID":"6e223b9a-8fcf-4870-8452-a001ff12f980","Type":"ContainerDied","Data":"3c7108aa875a803b226c93a15c517a317a1c30d631cfba0cd5496cf5168d620a"}
Nov 03 20:28:29.259228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:29.259149    5659 scope.go:122] "RemoveContainer" containerID="5bf6061272a9f2de0b35ad0465987214a947def1da00bc44606b047d783a0812"
Nov 03 20:28:29.259703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:29.259679    5659 scope.go:122] "RemoveContainer" containerID="3c7108aa875a803b226c93a15c517a317a1c30d631cfba0cd5496cf5168d620a"
Nov 03 20:28:29.260335 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:29.260196    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-fqdl5_odh-ai-gateway-infra(6e223b9a-8fcf-4870-8452-a001ff12f980)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5" podUID="6e223b9a-8fcf-4870-8452-a001ff12f980"
Nov 03 20:28:29.432599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:29.432503    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:29.836436 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:29.836400    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:29.975041 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:29.975008    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:30.164449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:30.164360    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:30.293463 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:30.293428    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"]
Nov 03 20:28:30.300852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:30.300819    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"]
Nov 03 20:28:30.326181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:30.326112    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:30.460266 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:30.460218    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:30.836512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:30.829715    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"
Nov 03 20:28:30.942527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:30.934149    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-xks-ca\") pod \"6e223b9a-8fcf-4870-8452-a001ff12f980\" (UID: \"6e223b9a-8fcf-4870-8452-a001ff12f980\") "
Nov 03 20:28:30.942527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:30.934254    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-openshift-ca\") pod \"6e223b9a-8fcf-4870-8452-a001ff12f980\" (UID: \"6e223b9a-8fcf-4870-8452-a001ff12f980\") "
Nov 03 20:28:30.942527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:30.934507    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e223b9a-8fcf-4870-8452-a001ff12f980-kube-api-access-8p4hz\" (UniqueName: \"kubernetes.io/projected/6e223b9a-8fcf-4870-8452-a001ff12f980-kube-api-access-8p4hz\") pod \"6e223b9a-8fcf-4870-8452-a001ff12f980\" (UID: \"6e223b9a-8fcf-4870-8452-a001ff12f980\") "
Nov 03 20:28:30.948351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:30.943622    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-openshift-ca" pod "6e223b9a-8fcf-4870-8452-a001ff12f980" (UID: "6e223b9a-8fcf-4870-8452-a001ff12f980"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:30.959214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:30.959175    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:31.065508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:31.044578    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:31.377664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:31.376946    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5" event={"ID":"6e223b9a-8fcf-4870-8452-a001ff12f980","Type":"ContainerDied","Data":"c0c016b3dcb490ae3d48ba0d9dcf1d6878d7f3dbf6afb7281f6b85673386c849"}
Nov 03 20:28:31.377664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:31.377033    5659 scope.go:122] "RemoveContainer" containerID="3c7108aa875a803b226c93a15c517a317a1c30d631cfba0cd5496cf5168d620a"
Nov 03 20:28:31.377664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:31.377155    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"
Nov 03 20:28:31.392834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:31.388918    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e223b9a-8fcf-4870-8452-a001ff12f980-kube-api-access-8p4hz" pod "6e223b9a-8fcf-4870-8452-a001ff12f980" (UID: "6e223b9a-8fcf-4870-8452-a001ff12f980"). InnerVolumeSpecName "kube-api-access-8p4hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:31.432744 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:31.429000    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-xks-ca" pod "6e223b9a-8fcf-4870-8452-a001ff12f980" (UID: "6e223b9a-8fcf-4870-8452-a001ff12f980"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:31.441179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:31.437571    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p4hz\" (UniqueName: \"kubernetes.io/projected/6e223b9a-8fcf-4870-8452-a001ff12f980-kube-api-access-8p4hz\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:31.441179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:31.437610    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e223b9a-8fcf-4870-8452-a001ff12f980-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:31.443474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:31.443444    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:31.710708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:31.710673    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"]
Nov 03 20:28:31.720176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:31.720141    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fqdl5"]
Nov 03 20:28:31.721135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:31.721103    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:31.973626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:31.973593    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e223b9a-8fcf-4870-8452-a001ff12f980" path="/var/lib/kubelet/pods/6e223b9a-8fcf-4870-8452-a001ff12f980/volumes"
Nov 03 20:28:32.020970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.020929    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"]
Nov 03 20:28:32.024683 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.024649    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c0273aaa-965c-4bc5-b34e-32e792396ead" containerName="revoke-keys"
Nov 03 20:28:32.024923 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.024907    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0273aaa-965c-4bc5-b34e-32e792396ead" containerName="revoke-keys"
Nov 03 20:28:32.025042 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.025030    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6e223b9a-8fcf-4870-8452-a001ff12f980" containerName="revoke-keys"
Nov 03 20:28:32.025126 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.025114    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e223b9a-8fcf-4870-8452-a001ff12f980" containerName="revoke-keys"
Nov 03 20:28:32.025230 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.025219    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6e223b9a-8fcf-4870-8452-a001ff12f980" containerName="revoke-keys"
Nov 03 20:28:32.025322 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.025308    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e223b9a-8fcf-4870-8452-a001ff12f980" containerName="revoke-keys"
Nov 03 20:28:32.027221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.026249    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e223b9a-8fcf-4870-8452-a001ff12f980" containerName="revoke-keys"
Nov 03 20:28:32.027634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.027616    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e223b9a-8fcf-4870-8452-a001ff12f980" containerName="revoke-keys"
Nov 03 20:28:32.027791 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.027778    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0273aaa-965c-4bc5-b34e-32e792396ead" containerName="revoke-keys"
Nov 03 20:28:32.045071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.045032    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"
Nov 03 20:28:32.054311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.054273    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"]
Nov 03 20:28:32.139578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.127456    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:32.160506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.156962    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-t5v8j\" (UID: \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"
Nov 03 20:28:32.161029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.160775    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px4js\" (UniqueName: \"kubernetes.io/projected/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-kube-api-access-px4js\") pod \"maas-api-revoke-sub-abad4031-t5v8j\" (UID: \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"
Nov 03 20:28:32.161300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.161265    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-t5v8j\" (UID: \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"
Nov 03 20:28:32.262755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.262657    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-t5v8j\" (UID: \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"
Nov 03 20:28:32.262928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.262833    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px4js\" (UniqueName: \"kubernetes.io/projected/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-kube-api-access-px4js\") pod \"maas-api-revoke-sub-abad4031-t5v8j\" (UID: \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"
Nov 03 20:28:32.263308 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.263129    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-t5v8j\" (UID: \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"
Nov 03 20:28:32.263538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.263514    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-t5v8j\" (UID: \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"
Nov 03 20:28:32.265737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.265705    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-t5v8j\" (UID: \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"
Nov 03 20:28:32.275193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.275168    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px4js\" (UniqueName: \"kubernetes.io/projected/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-kube-api-access-px4js\") pod \"maas-api-revoke-sub-abad4031-t5v8j\" (UID: \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"
Nov 03 20:28:32.369858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.369812    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"
Nov 03 20:28:32.662470 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:28:32.662437    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4bbe94_eacd_4942_ba57_84b0e5d7f2cf.slice/crio-8ef17a9cf183501c4f669fadc66b91ce40c252e551b49acc7be3a9e5d5d84f36 WatchSource:0}: Error finding container 8ef17a9cf183501c4f669fadc66b91ce40c252e551b49acc7be3a9e5d5d84f36: Status 404 returned error can't find the container with id 8ef17a9cf183501c4f669fadc66b91ce40c252e551b49acc7be3a9e5d5d84f36
Nov 03 20:28:32.663539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.663515    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"]
Nov 03 20:28:32.754643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:32.754604    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:33.402166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:33.402127    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j" event={"ID":"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf","Type":"ContainerStarted","Data":"3bf850d37c1539f6939ea30f166d5abeeafd967d5fcfc8c3ccb396e92b953090"}
Nov 03 20:28:33.402672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:33.402194    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j" event={"ID":"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf","Type":"ContainerStarted","Data":"8ef17a9cf183501c4f669fadc66b91ce40c252e551b49acc7be3a9e5d5d84f36"}
Nov 03 20:28:33.425501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:33.425443    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j" podStartSLOduration=2.425429506 podStartE2EDuration="2.425429506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:33.419857346 +0000 UTC m=+3065.574759767" watchObservedRunningTime="2026-11-03 20:28:33.425429506 +0000 UTC m=+3065.580331924"
Nov 03 20:28:33.484154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:33.484121    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:33.628507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:33.623382    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:34.411759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:34.411721    5659 generic.go:356] "Generic (PLEG): container finished" podID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" containerID="3bf850d37c1539f6939ea30f166d5abeeafd967d5fcfc8c3ccb396e92b953090" exitCode=22
Nov 03 20:28:34.412173 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:34.411799    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j" event={"ID":"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf","Type":"ContainerDied","Data":"3bf850d37c1539f6939ea30f166d5abeeafd967d5fcfc8c3ccb396e92b953090"}
Nov 03 20:28:34.412173 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:34.412093    5659 scope.go:122] "RemoveContainer" containerID="3bf850d37c1539f6939ea30f166d5abeeafd967d5fcfc8c3ccb396e92b953090"
Nov 03 20:28:34.438755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:34.438726    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:35.155558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:35.153594    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:35.294389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:35.294346    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k" podUID="645b8d2f-2118-4686-b386-4e90a7842a08" containerName="payload-pre-processing" containerID="cri-o://bb85250666cf25e370089e34276c00007af0e6fabc1663b22ae55a1360365b6c" gracePeriod=25
Nov 03 20:28:35.423147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:35.423105    5659 generic.go:356] "Generic (PLEG): container finished" podID="645b8d2f-2118-4686-b386-4e90a7842a08" containerID="bb85250666cf25e370089e34276c00007af0e6fabc1663b22ae55a1360365b6c" exitCode=0
Nov 03 20:28:35.423692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:35.423172    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k" event={"ID":"645b8d2f-2118-4686-b386-4e90a7842a08","Type":"ContainerDied","Data":"bb85250666cf25e370089e34276c00007af0e6fabc1663b22ae55a1360365b6c"}
Nov 03 20:28:35.425702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:35.425671    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j" event={"ID":"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf","Type":"ContainerStarted","Data":"df8f7aa560e93e862bd37142fa1e17387ee97bfc9c80e3b36ce572e24ab36b19"}
Nov 03 20:28:35.674231 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:35.673521    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"
Nov 03 20:28:35.737857 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:35.737813    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/645b8d2f-2118-4686-b386-4e90a7842a08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b8d2f-2118-4686-b386-4e90a7842a08-plugins-config-volume\") pod \"645b8d2f-2118-4686-b386-4e90a7842a08\" (UID: \"645b8d2f-2118-4686-b386-4e90a7842a08\") "
Nov 03 20:28:35.738352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:35.738241    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645b8d2f-2118-4686-b386-4e90a7842a08-plugins-config-volume" pod "645b8d2f-2118-4686-b386-4e90a7842a08" (UID: "645b8d2f-2118-4686-b386-4e90a7842a08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:35.738533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:35.738449    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645b8d2f-2118-4686-b386-4e90a7842a08-kube-api-access-x2tqn\" (UniqueName: \"kubernetes.io/projected/645b8d2f-2118-4686-b386-4e90a7842a08-kube-api-access-x2tqn\") pod \"645b8d2f-2118-4686-b386-4e90a7842a08\" (UID: \"645b8d2f-2118-4686-b386-4e90a7842a08\") "
Nov 03 20:28:35.739683 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:35.739659    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b8d2f-2118-4686-b386-4e90a7842a08-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:35.747522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:35.747001    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645b8d2f-2118-4686-b386-4e90a7842a08-kube-api-access-x2tqn" pod "645b8d2f-2118-4686-b386-4e90a7842a08" (UID: "645b8d2f-2118-4686-b386-4e90a7842a08"). InnerVolumeSpecName "kube-api-access-x2tqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:35.841724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:35.841692    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2tqn\" (UniqueName: \"kubernetes.io/projected/645b8d2f-2118-4686-b386-4e90a7842a08-kube-api-access-x2tqn\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:35.899091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:35.899054    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:36.438201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:36.438108    5659 generic.go:356] "Generic (PLEG): container finished" podID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" containerID="df8f7aa560e93e862bd37142fa1e17387ee97bfc9c80e3b36ce572e24ab36b19" exitCode=22
Nov 03 20:28:36.438770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:36.438342    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j" event={"ID":"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf","Type":"ContainerDied","Data":"df8f7aa560e93e862bd37142fa1e17387ee97bfc9c80e3b36ce572e24ab36b19"}
Nov 03 20:28:36.438770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:36.438437    5659 scope.go:122] "RemoveContainer" containerID="3bf850d37c1539f6939ea30f166d5abeeafd967d5fcfc8c3ccb396e92b953090"
Nov 03 20:28:36.439269 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:36.439235    5659 scope.go:122] "RemoveContainer" containerID="df8f7aa560e93e862bd37142fa1e17387ee97bfc9c80e3b36ce572e24ab36b19"
Nov 03 20:28:36.440280 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:36.439726    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-t5v8j_odh-ai-gateway-infra(ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j" podUID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf"
Nov 03 20:28:36.441364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:36.440616    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k" event={"ID":"645b8d2f-2118-4686-b386-4e90a7842a08","Type":"ContainerDied","Data":"04a81202b63f2bddf2920ca1bec55ba31d2ec700f5bb18437f5d91f81c1c53c1"}
Nov 03 20:28:36.441364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:36.440666    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"
Nov 03 20:28:36.455233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:36.455206    5659 scope.go:122] "RemoveContainer" containerID="bb85250666cf25e370089e34276c00007af0e6fabc1663b22ae55a1360365b6c"
Nov 03 20:28:36.476981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:36.476948    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"]
Nov 03 20:28:36.483362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:36.483335    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d2aa3b-6c4ff9988c-stg9k"]
Nov 03 20:28:36.638259 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:36.638196    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:37.329279 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:37.329245    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:37.500371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:37.495641    5659 scope.go:122] "RemoveContainer" containerID="df8f7aa560e93e862bd37142fa1e17387ee97bfc9c80e3b36ce572e24ab36b19"
Nov 03 20:28:37.500371 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:37.496215    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-t5v8j_odh-ai-gateway-infra(ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j" podUID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf"
Nov 03 20:28:37.850263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:37.850227    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:37.972833 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:37.972798    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645b8d2f-2118-4686-b386-4e90a7842a08" path="/var/lib/kubelet/pods/645b8d2f-2118-4686-b386-4e90a7842a08/volumes"
Nov 03 20:28:38.060785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:38.060751    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:38.555060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:38.554986    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:38.827375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:38.827249    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:38.961422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:38.961391    5659 scope.go:122] "RemoveContainer" containerID="333551521d0521be39f9cf1ff29055c135d0502a4be49032c035d0a3b27bdbb4"
Nov 03 20:28:39.521982 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:39.521868    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:39.530309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:39.530274    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb" event={"ID":"56aabd38-0e31-44ed-b489-f7eb2fc4e430","Type":"ContainerStarted","Data":"fc29ee41ab7aa4386802531b35f968cade984f3fc8d2662832c0bb59772a96fc"}
Nov 03 20:28:40.253690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:40.253655    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:40.543664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:40.543568    5659 generic.go:356] "Generic (PLEG): container finished" podID="56aabd38-0e31-44ed-b489-f7eb2fc4e430" containerID="fc29ee41ab7aa4386802531b35f968cade984f3fc8d2662832c0bb59772a96fc" exitCode=22
Nov 03 20:28:40.543664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:40.543600    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb" event={"ID":"56aabd38-0e31-44ed-b489-f7eb2fc4e430","Type":"ContainerDied","Data":"fc29ee41ab7aa4386802531b35f968cade984f3fc8d2662832c0bb59772a96fc"}
Nov 03 20:28:40.543906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:40.543708    5659 scope.go:122] "RemoveContainer" containerID="333551521d0521be39f9cf1ff29055c135d0502a4be49032c035d0a3b27bdbb4"
Nov 03 20:28:40.544326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:40.544300    5659 scope.go:122] "RemoveContainer" containerID="fc29ee41ab7aa4386802531b35f968cade984f3fc8d2662832c0bb59772a96fc"
Nov 03 20:28:40.545092 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:40.544814    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-wf2rb_odh-ai-gateway-infra(56aabd38-0e31-44ed-b489-f7eb2fc4e430)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb" podUID="56aabd38-0e31-44ed-b489-f7eb2fc4e430"
Nov 03 20:28:40.563447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:40.563416    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"]
Nov 03 20:28:40.940154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:40.940048    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:41.738520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:41.738474    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:41.823578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:41.823504    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"
Nov 03 20:28:41.917645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:41.917608    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-xks-ca\") pod \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\" (UID: \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\") "
Nov 03 20:28:41.917865 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:41.917727    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56aabd38-0e31-44ed-b489-f7eb2fc4e430-kube-api-access-n7x7s\" (UniqueName: \"kubernetes.io/projected/56aabd38-0e31-44ed-b489-f7eb2fc4e430-kube-api-access-n7x7s\") pod \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\" (UID: \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\") "
Nov 03 20:28:41.918268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:41.917952    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-openshift-ca\") pod \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\" (UID: \"56aabd38-0e31-44ed-b489-f7eb2fc4e430\") "
Nov 03 20:28:41.918447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:41.918399    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-openshift-ca" pod "56aabd38-0e31-44ed-b489-f7eb2fc4e430" (UID: "56aabd38-0e31-44ed-b489-f7eb2fc4e430"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:41.919078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:41.919052    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:41.921230 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:41.921205    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56aabd38-0e31-44ed-b489-f7eb2fc4e430-kube-api-access-n7x7s" pod "56aabd38-0e31-44ed-b489-f7eb2fc4e430" (UID: "56aabd38-0e31-44ed-b489-f7eb2fc4e430"). InnerVolumeSpecName "kube-api-access-n7x7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:41.926851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:41.926824    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-xks-ca" pod "56aabd38-0e31-44ed-b489-f7eb2fc4e430" (UID: "56aabd38-0e31-44ed-b489-f7eb2fc4e430"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:41.982715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:41.982685    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:42.020355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:42.020324    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56aabd38-0e31-44ed-b489-f7eb2fc4e430-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:42.020355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:42.020353    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7x7s\" (UniqueName: \"kubernetes.io/projected/56aabd38-0e31-44ed-b489-f7eb2fc4e430-kube-api-access-n7x7s\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:42.568518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:42.568459    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"
Nov 03 20:28:42.568728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:42.568528    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb" event={"ID":"56aabd38-0e31-44ed-b489-f7eb2fc4e430","Type":"ContainerDied","Data":"00a9bb0aa373cd4b04fdc6d8aa910b336d05bdd7573b8eb48502048a406afc8b"}
Nov 03 20:28:42.568728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:42.568596    5659 scope.go:122] "RemoveContainer" containerID="fc29ee41ab7aa4386802531b35f968cade984f3fc8d2662832c0bb59772a96fc"
Nov 03 20:28:42.593063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:42.593039    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"]
Nov 03 20:28:42.601356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:42.601331    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-wf2rb"]
Nov 03 20:28:42.627121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:42.627091    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:42.998879 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:42.998851    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:43.267603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:43.267475    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:43.972379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:43.972346    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56aabd38-0e31-44ed-b489-f7eb2fc4e430" path="/var/lib/kubelet/pods/56aabd38-0e31-44ed-b489-f7eb2fc4e430/volumes"
Nov 03 20:28:44.064098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:44.064060    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:44.786460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:44.786418    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:45.219534 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:45.219457    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:28:45.443221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:45.443188    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:46.316345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:46.316314    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:47.098924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:47.098887    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:47.794684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:47.794630    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:48.534100 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:48.534065    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:49.351337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:49.351297    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:49.962084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:49.962045    5659 scope.go:122] "RemoveContainer" containerID="df8f7aa560e93e862bd37142fa1e17387ee97bfc9c80e3b36ce572e24ab36b19"
Nov 03 20:28:50.036679 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:50.036596    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:50.697999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:50.697908    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j" event={"ID":"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf","Type":"ContainerStarted","Data":"ab78ea46d9d47dd22710fccec763141787becb360c5a1934db7edf3a68549d7c"}
Nov 03 20:28:50.870215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:50.870178    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:51.640951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:51.640917    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:51.708748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:51.708713    5659 generic.go:356] "Generic (PLEG): container finished" podID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" containerID="ab78ea46d9d47dd22710fccec763141787becb360c5a1934db7edf3a68549d7c" exitCode=22
Nov 03 20:28:51.709242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:51.708784    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j" event={"ID":"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf","Type":"ContainerDied","Data":"ab78ea46d9d47dd22710fccec763141787becb360c5a1934db7edf3a68549d7c"}
Nov 03 20:28:51.709242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:51.708845    5659 scope.go:122] "RemoveContainer" containerID="df8f7aa560e93e862bd37142fa1e17387ee97bfc9c80e3b36ce572e24ab36b19"
Nov 03 20:28:51.709368 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:51.709349    5659 scope.go:122] "RemoveContainer" containerID="ab78ea46d9d47dd22710fccec763141787becb360c5a1934db7edf3a68549d7c"
Nov 03 20:28:51.709871 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:51.709828    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-t5v8j_odh-ai-gateway-infra(ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j" podUID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf"
Nov 03 20:28:51.748375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:51.748348    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"]
Nov 03 20:28:52.074557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.074467    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"]
Nov 03 20:28:52.119366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.119240    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:52.442639 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.428447    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:52.458066 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.458025    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"]
Nov 03 20:28:52.462546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.459569    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6e223b9a-8fcf-4870-8452-a001ff12f980" containerName="revoke-keys"
Nov 03 20:28:52.462770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.462746    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e223b9a-8fcf-4870-8452-a001ff12f980" containerName="revoke-keys"
Nov 03 20:28:52.462906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.462895    5659 cpu_manager.go:414] "containerMap: removing container" podUID="56aabd38-0e31-44ed-b489-f7eb2fc4e430" containerName="revoke-keys"
Nov 03 20:28:52.462989 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.462978    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56aabd38-0e31-44ed-b489-f7eb2fc4e430" containerName="revoke-keys"
Nov 03 20:28:52.463095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.463084    5659 cpu_manager.go:414] "containerMap: removing container" podUID="645b8d2f-2118-4686-b386-4e90a7842a08" containerName="payload-pre-processing"
Nov 03 20:28:52.463174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.463164    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645b8d2f-2118-4686-b386-4e90a7842a08" containerName="payload-pre-processing"
Nov 03 20:28:52.463358 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.463344    5659 cpu_manager.go:414] "containerMap: removing container" podUID="56aabd38-0e31-44ed-b489-f7eb2fc4e430" containerName="revoke-keys"
Nov 03 20:28:52.463455 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.463444    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56aabd38-0e31-44ed-b489-f7eb2fc4e430" containerName="revoke-keys"
Nov 03 20:28:52.463583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.463571    5659 cpu_manager.go:414] "containerMap: removing container" podUID="56aabd38-0e31-44ed-b489-f7eb2fc4e430" containerName="revoke-keys"
Nov 03 20:28:52.463659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.463650    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56aabd38-0e31-44ed-b489-f7eb2fc4e430" containerName="revoke-keys"
Nov 03 20:28:52.463994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.463977    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="645b8d2f-2118-4686-b386-4e90a7842a08" containerName="payload-pre-processing"
Nov 03 20:28:52.464130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.464116    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="56aabd38-0e31-44ed-b489-f7eb2fc4e430" containerName="revoke-keys"
Nov 03 20:28:52.464234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.464222    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="56aabd38-0e31-44ed-b489-f7eb2fc4e430" containerName="revoke-keys"
Nov 03 20:28:52.464339 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.464328    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e223b9a-8fcf-4870-8452-a001ff12f980" containerName="revoke-keys"
Nov 03 20:28:52.472533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.472511    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"
Nov 03 20:28:52.482758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.482721    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"]
Nov 03 20:28:52.488178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.488143    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57807933-a9af-459d-a489-698687afaa45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-6d22b\" (UID: \"57807933-a9af-459d-a489-698687afaa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"
Nov 03 20:28:52.488569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.488549    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vzh\" (UniqueName: \"kubernetes.io/projected/57807933-a9af-459d-a489-698687afaa45-kube-api-access-47vzh\") pod \"maas-api-revoke-sub-5bac6f1e-6d22b\" (UID: \"57807933-a9af-459d-a489-698687afaa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"
Nov 03 20:28:52.488769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.488752    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57807933-a9af-459d-a489-698687afaa45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-6d22b\" (UID: \"57807933-a9af-459d-a489-698687afaa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"
Nov 03 20:28:52.590905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.590869    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vzh\" (UniqueName: \"kubernetes.io/projected/57807933-a9af-459d-a489-698687afaa45-kube-api-access-47vzh\") pod \"maas-api-revoke-sub-5bac6f1e-6d22b\" (UID: \"57807933-a9af-459d-a489-698687afaa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"
Nov 03 20:28:52.591229 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.591207    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57807933-a9af-459d-a489-698687afaa45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-6d22b\" (UID: \"57807933-a9af-459d-a489-698687afaa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"
Nov 03 20:28:52.591697 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.591672    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57807933-a9af-459d-a489-698687afaa45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-6d22b\" (UID: \"57807933-a9af-459d-a489-698687afaa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"
Nov 03 20:28:52.592852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.592831    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57807933-a9af-459d-a489-698687afaa45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-6d22b\" (UID: \"57807933-a9af-459d-a489-698687afaa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"
Nov 03 20:28:52.660614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.647597    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vzh\" (UniqueName: \"kubernetes.io/projected/57807933-a9af-459d-a489-698687afaa45-kube-api-access-47vzh\") pod \"maas-api-revoke-sub-5bac6f1e-6d22b\" (UID: \"57807933-a9af-459d-a489-698687afaa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"
Nov 03 20:28:52.671503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.669947    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57807933-a9af-459d-a489-698687afaa45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-6d22b\" (UID: \"57807933-a9af-459d-a489-698687afaa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"
Nov 03 20:28:52.724557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.724475    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr" event={"ID":"59b60f28-7800-45ba-9fa6-3964633c5df3","Type":"ContainerDied","Data":"156cce8c215182bafa50a7aa6f6a5f50bf10b426db96bdb88172ceeb1c130836"}
Nov 03 20:28:52.725020 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.724583    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="156cce8c215182bafa50a7aa6f6a5f50bf10b426db96bdb88172ceeb1c130836"
Nov 03 20:28:52.753956 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.753926    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:28:52.796639 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.796604    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b60f28-7800-45ba-9fa6-3964633c5df3-kube-api-access-kwsd6\" (UniqueName: \"kubernetes.io/projected/59b60f28-7800-45ba-9fa6-3964633c5df3-kube-api-access-kwsd6\") pod \"59b60f28-7800-45ba-9fa6-3964633c5df3\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") "
Nov 03 20:28:52.796799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.796749    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-maas-api-tls\") pod \"59b60f28-7800-45ba-9fa6-3964633c5df3\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") "
Nov 03 20:28:52.796965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.796945    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-metrics-certs\") pod \"59b60f28-7800-45ba-9fa6-3964633c5df3\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") "
Nov 03 20:28:52.797079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.797063    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b60f28-7800-45ba-9fa6-3964633c5df3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b60f28-7800-45ba-9fa6-3964633c5df3-openshift-service-ca\") pod \"59b60f28-7800-45ba-9fa6-3964633c5df3\" (UID: \"59b60f28-7800-45ba-9fa6-3964633c5df3\") "
Nov 03 20:28:52.802899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.802868    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b60f28-7800-45ba-9fa6-3964633c5df3-openshift-service-ca" pod "59b60f28-7800-45ba-9fa6-3964633c5df3" (UID: "59b60f28-7800-45ba-9fa6-3964633c5df3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:52.804844 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.804818    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-maas-api-tls" pod "59b60f28-7800-45ba-9fa6-3964633c5df3" (UID: "59b60f28-7800-45ba-9fa6-3964633c5df3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:52.809752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.809712    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-metrics-certs" pod "59b60f28-7800-45ba-9fa6-3964633c5df3" (UID: "59b60f28-7800-45ba-9fa6-3964633c5df3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:52.816125 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.816032    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b60f28-7800-45ba-9fa6-3964633c5df3-kube-api-access-kwsd6" pod "59b60f28-7800-45ba-9fa6-3964633c5df3" (UID: "59b60f28-7800-45ba-9fa6-3964633c5df3"). InnerVolumeSpecName "kube-api-access-kwsd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:52.884845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.884817    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"
Nov 03 20:28:52.897510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.895021    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"
Nov 03 20:28:52.905278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.905234    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-kube-api-access-px4js\" (UniqueName: \"kubernetes.io/projected/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-kube-api-access-px4js\") pod \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\" (UID: \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\") "
Nov 03 20:28:52.905713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.905684    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-xks-ca\") pod \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\" (UID: \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\") "
Nov 03 20:28:52.905841 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.905762    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-openshift-ca\") pod \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\" (UID: \"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf\") "
Nov 03 20:28:52.906722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.906698    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:52.906958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.906727    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59b60f28-7800-45ba-9fa6-3964633c5df3-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:52.906958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.906744    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b60f28-7800-45ba-9fa6-3964633c5df3-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:52.906958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.906760    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwsd6\" (UniqueName: \"kubernetes.io/projected/59b60f28-7800-45ba-9fa6-3964633c5df3-kube-api-access-kwsd6\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:52.907320 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.907147    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-openshift-ca" pod "ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" (UID: "ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:52.926190 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.923408    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-kube-api-access-px4js" pod "ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" (UID: "ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf"). InnerVolumeSpecName "kube-api-access-px4js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:52.927976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:52.927944    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-xks-ca" pod "ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" (UID: "ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:53.011118 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.011083    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:53.011118 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.011118    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px4js\" (UniqueName: \"kubernetes.io/projected/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-kube-api-access-px4js\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:53.011377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.011132    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:53.153760 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.153730    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:53.260767 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:28:53.260719    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57807933_a9af_459d_a489_698687afaa45.slice/crio-9fe1635c612b3f8e8b4564f57d43ed48531358fbeeaa27a9312fd3884874d74e WatchSource:0}: Error finding container 9fe1635c612b3f8e8b4564f57d43ed48531358fbeeaa27a9312fd3884874d74e: Status 404 returned error can't find the container with id 9fe1635c612b3f8e8b4564f57d43ed48531358fbeeaa27a9312fd3884874d74e
Nov 03 20:28:53.263638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.263608    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"]
Nov 03 20:28:53.734317 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.734280    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"
Nov 03 20:28:53.735101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.734288    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j" event={"ID":"ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf","Type":"ContainerDied","Data":"8ef17a9cf183501c4f669fadc66b91ce40c252e551b49acc7be3a9e5d5d84f36"}
Nov 03 20:28:53.735101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.734464    5659 scope.go:122] "RemoveContainer" containerID="ab78ea46d9d47dd22710fccec763141787becb360c5a1934db7edf3a68549d7c"
Nov 03 20:28:53.736308 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.736282    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b" event={"ID":"57807933-a9af-459d-a489-698687afaa45","Type":"ContainerStarted","Data":"9fe1635c612b3f8e8b4564f57d43ed48531358fbeeaa27a9312fd3884874d74e"}
Nov 03 20:28:53.736418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.736400    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"
Nov 03 20:28:53.767062 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.767030    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"]
Nov 03 20:28:53.771806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.771778    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d2aa3b-5bcf9467c8-jmwpr"]
Nov 03 20:28:53.784912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.784872    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"]
Nov 03 20:28:53.791424 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.791394    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-t5v8j"]
Nov 03 20:28:53.973982 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.973948    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" path="/var/lib/kubelet/pods/59b60f28-7800-45ba-9fa6-3964633c5df3/volumes"
Nov 03 20:28:53.974538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:53.974521    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" path="/var/lib/kubelet/pods/ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf/volumes"
Nov 03 20:28:54.023309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:54.023280    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:54.156155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:54.156125    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:54.521510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:54.512821    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:54.753429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:54.753388    5659 generic.go:356] "Generic (PLEG): container finished" podID="57807933-a9af-459d-a489-698687afaa45" containerID="82d2b55f9edc713e36e690de3d09861faa668d9f9a619354f0d6f21cac25120a" exitCode=22
Nov 03 20:28:54.753939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:54.753786    5659 scope.go:122] "RemoveContainer" containerID="82d2b55f9edc713e36e690de3d09861faa668d9f9a619354f0d6f21cac25120a"
Nov 03 20:28:54.754053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:54.754024    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b" event={"ID":"57807933-a9af-459d-a489-698687afaa45","Type":"ContainerDied","Data":"82d2b55f9edc713e36e690de3d09861faa668d9f9a619354f0d6f21cac25120a"}
Nov 03 20:28:54.853801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:54.853774    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:55.548337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:55.548284    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:55.766294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:55.766253    5659 generic.go:356] "Generic (PLEG): container finished" podID="57807933-a9af-459d-a489-698687afaa45" containerID="8957882e014366dff43a87f9bbb26177b8623186e7bae6d9c9b760c4133c18e1" exitCode=22
Nov 03 20:28:55.766803 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:55.766538    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b" event={"ID":"57807933-a9af-459d-a489-698687afaa45","Type":"ContainerDied","Data":"8957882e014366dff43a87f9bbb26177b8623186e7bae6d9c9b760c4133c18e1"}
Nov 03 20:28:55.766803 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:55.766609    5659 scope.go:122] "RemoveContainer" containerID="82d2b55f9edc713e36e690de3d09861faa668d9f9a619354f0d6f21cac25120a"
Nov 03 20:28:55.766803 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:55.766749    5659 scope.go:122] "RemoveContainer" containerID="8957882e014366dff43a87f9bbb26177b8623186e7bae6d9c9b760c4133c18e1"
Nov 03 20:28:55.767216 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:55.767186    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-6d22b_odh-ai-gateway-infra(57807933-a9af-459d-a489-698687afaa45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b" podUID="57807933-a9af-459d-a489-698687afaa45"
Nov 03 20:28:56.336504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:56.336020    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:56.778154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:56.778122    5659 scope.go:122] "RemoveContainer" containerID="8957882e014366dff43a87f9bbb26177b8623186e7bae6d9c9b760c4133c18e1"
Nov 03 20:28:56.778692 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:28:56.778590    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-6d22b_odh-ai-gateway-infra(57807933-a9af-459d-a489-698687afaa45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b" podUID="57807933-a9af-459d-a489-698687afaa45"
Nov 03 20:28:57.085569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:57.072997    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:57.830040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:57.829996    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:58.535456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:58.535411    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:59.325546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:28:59.325516    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:00.139666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:00.139630    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:00.838401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:00.838366    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:00.967541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:00.967474    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:01.812999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:01.812965    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:02.053059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.053027    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"]
Nov 03 20:29:02.053997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.053981    5659 cpu_manager.go:414] "containerMap: removing container" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.054071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.053999    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.054071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054021    5659 cpu_manager.go:414] "containerMap: removing container" podUID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" containerName="revoke-keys"
Nov 03 20:29:02.054071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054026    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" containerName="revoke-keys"
Nov 03 20:29:02.054071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054040    5659 cpu_manager.go:414] "containerMap: removing container" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.054071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054044    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.054071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054051    5659 cpu_manager.go:414] "containerMap: removing container" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.054071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054055    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.054071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054073    5659 cpu_manager.go:414] "containerMap: removing container" podUID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" containerName="revoke-keys"
Nov 03 20:29:02.054380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054079    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" containerName="revoke-keys"
Nov 03 20:29:02.054380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054095    5659 cpu_manager.go:414] "containerMap: removing container" podUID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" containerName="revoke-keys"
Nov 03 20:29:02.054380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054101    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" containerName="revoke-keys"
Nov 03 20:29:02.054380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054119    5659 cpu_manager.go:414] "containerMap: removing container" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.054380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054124    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.054380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054277    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.054380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054292    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" containerName="revoke-keys"
Nov 03 20:29:02.054380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054307    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.054380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054314    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.054380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054322    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" containerName="revoke-keys"
Nov 03 20:29:02.054380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054329    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4bbe94-eacd-4942-ba57-84b0e5d7f2cf" containerName="revoke-keys"
Nov 03 20:29:02.054380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054338    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.054380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.054347    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="56aabd38-0e31-44ed-b489-f7eb2fc4e430" containerName="revoke-keys"
Nov 03 20:29:02.064018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.063966    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"
Nov 03 20:29:02.071939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.071854    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"]
Nov 03 20:29:02.143562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.143512    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-lv7jw\" (UID: \"96cfde48-50fa-46b8-8d5f-645f54101110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"
Nov 03 20:29:02.143804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.143682    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk9bw\" (UniqueName: \"kubernetes.io/projected/96cfde48-50fa-46b8-8d5f-645f54101110-kube-api-access-hk9bw\") pod \"maas-api-revoke-sub-abad4031-lv7jw\" (UID: \"96cfde48-50fa-46b8-8d5f-645f54101110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"
Nov 03 20:29:02.143882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.143834    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-lv7jw\" (UID: \"96cfde48-50fa-46b8-8d5f-645f54101110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"
Nov 03 20:29:02.245976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.245937    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-lv7jw\" (UID: \"96cfde48-50fa-46b8-8d5f-645f54101110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"
Nov 03 20:29:02.246160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.246002    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9bw\" (UniqueName: \"kubernetes.io/projected/96cfde48-50fa-46b8-8d5f-645f54101110-kube-api-access-hk9bw\") pod \"maas-api-revoke-sub-abad4031-lv7jw\" (UID: \"96cfde48-50fa-46b8-8d5f-645f54101110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"
Nov 03 20:29:02.246160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.246036    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-lv7jw\" (UID: \"96cfde48-50fa-46b8-8d5f-645f54101110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"
Nov 03 20:29:02.256038 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.247227    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-lv7jw\" (UID: \"96cfde48-50fa-46b8-8d5f-645f54101110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"
Nov 03 20:29:02.256038 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.249500    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-lv7jw\" (UID: \"96cfde48-50fa-46b8-8d5f-645f54101110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"
Nov 03 20:29:02.260272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.260233    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk9bw\" (UniqueName: \"kubernetes.io/projected/96cfde48-50fa-46b8-8d5f-645f54101110-kube-api-access-hk9bw\") pod \"maas-api-revoke-sub-abad4031-lv7jw\" (UID: \"96cfde48-50fa-46b8-8d5f-645f54101110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"
Nov 03 20:29:02.380733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.380655    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"
Nov 03 20:29:02.416912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.416882    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:02.485665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.481246    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"]
Nov 03 20:29:02.485665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.483426    5659 cpu_manager.go:414] "containerMap: removing container" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.485665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.483451    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.485665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.483694    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b60f28-7800-45ba-9fa6-3964633c5df3" containerName="maas-api"
Nov 03 20:29:02.597559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.596711    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"
Nov 03 20:29:02.603211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.602070    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bfvcm\""
Nov 03 20:29:02.610190 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.610158    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 20:29:02.610398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.610353    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 20:29:02.610470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.610438    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:02.616056 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.612201    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:02.616056 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.612282    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mnp77"]
Nov 03 20:29:02.625722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.623609    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77"
Nov 03 20:29:02.628848 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.627555    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"]
Nov 03 20:29:02.641246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.641216    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"]
Nov 03 20:29:02.641688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.641667    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"
Nov 03 20:29:02.649965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.648267    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"
Nov 03 20:29:02.650625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.650514    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"]
Nov 03 20:29:02.657703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.657678    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:02.659387 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.659363    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"]
Nov 03 20:29:02.673067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.670687    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mnp77"]
Nov 03 20:29:02.684619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.684582    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"]
Nov 03 20:29:02.690542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.690199    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"]
Nov 03 20:29:02.694642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.694617    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"]
Nov 03 20:29:02.699128 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.699103    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pqq2n"]
Nov 03 20:29:02.702608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.702313    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 20:29:02.702608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.702342    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fw8qp\""
Nov 03 20:29:02.702608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.702395    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 20:29:02.702608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.702442    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 20:29:02.702608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.702573    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-p6m7s\""
Nov 03 20:29:02.703751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.703729    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-b22jw\""
Nov 03 20:29:02.703864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.703805    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rwt79\""
Nov 03 20:29:02.703943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.703866    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 20:29:02.703943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.703925    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 20:29:02.704328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.704307    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pqq2n"
Nov 03 20:29:02.704587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.704560    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmk2h\" (UniqueName: \"kubernetes.io/projected/4d4bda65-68f7-4c52-95ff-b0d7911b05f1-kube-api-access-dmk2h\") pod \"service-ca-operator-544d749c8c-mgfzm\" (UID: \"4d4bda65-68f7-4c52-95ff-b0d7911b05f1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"
Nov 03 20:29:02.704753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.704738    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d4bda65-68f7-4c52-95ff-b0d7911b05f1-config\") pod \"service-ca-operator-544d749c8c-mgfzm\" (UID: \"4d4bda65-68f7-4c52-95ff-b0d7911b05f1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"
Nov 03 20:29:02.704839 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.704811    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d4bda65-68f7-4c52-95ff-b0d7911b05f1-serving-cert\") pod \"service-ca-operator-544d749c8c-mgfzm\" (UID: \"4d4bda65-68f7-4c52-95ff-b0d7911b05f1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"
Nov 03 20:29:02.705180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.705159    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pqq2n"]
Nov 03 20:29:02.710360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.710332    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:02.710547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.710512    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 20:29:02.710547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.710534    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 20:29:02.710690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.710584    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 20:29:02.710690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.710663    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 20:29:02.710690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.710671    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 20:29:02.710859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.710732    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 20:29:02.711016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.710996    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 20:29:02.711524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.711503    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:02.711850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.711829    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4p6h4\""
Nov 03 20:29:02.711946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.711867    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:02.711946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.711934    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 20:29:02.712070 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.711939    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 20:29:02.712070 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.712009    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-8dg79"]
Nov 03 20:29:02.714648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.714622    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 20:29:02.718785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.718373    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"]
Nov 03 20:29:02.718785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.718546    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-8dg79"
Nov 03 20:29:02.728516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.727000    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"]
Nov 03 20:29:02.728516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.727128    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"
Nov 03 20:29:02.730805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.730532    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-8dg79"]
Nov 03 20:29:02.802034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.802004    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-wftlv\""
Nov 03 20:29:02.802034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.802004    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ss8kd\""
Nov 03 20:29:02.802255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.802073    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 20:29:02.806477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.806452    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5tnw\" (UniqueName: \"kubernetes.io/projected/419f6a62-6603-4ad3-b1b1-144adae9ffca-kube-api-access-w5tnw\") pod \"migrator-66df5d496b-pqq2n\" (UID: \"419f6a62-6603-4ad3-b1b1-144adae9ffca\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pqq2n"
Nov 03 20:29:02.806633 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.806523    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a72b034-13ca-4e05-babf-97728947b2ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6tkgl\" (UID: \"0a72b034-13ca-4e05-babf-97728947b2ac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"
Nov 03 20:29:02.806633 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.806612    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hs7j\" (UniqueName: \"kubernetes.io/projected/0a72b034-13ca-4e05-babf-97728947b2ac-kube-api-access-2hs7j\") pod \"cluster-monitoring-operator-884576c57-6tkgl\" (UID: \"0a72b034-13ca-4e05-babf-97728947b2ac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"
Nov 03 20:29:02.806727 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.806664    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01ae1c4c-fc05-473f-ae47-078e18d88e14-signing-key\") pod \"service-ca-5fffc97bdf-mnp77\" (UID: \"01ae1c4c-fc05-473f-ae47-078e18d88e14\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77"
Nov 03 20:29:02.806890 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.806865    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/764bdfb5-060a-4657-b9e0-254f1795eea8-serving-cert\") pod \"console-operator-7cd56b8dbc-6k7gv\" (UID: \"764bdfb5-060a-4657-b9e0-254f1795eea8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:02.806941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.806919    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbgqs\" (UniqueName: \"kubernetes.io/projected/01ae1c4c-fc05-473f-ae47-078e18d88e14-kube-api-access-wbgqs\") pod \"service-ca-5fffc97bdf-mnp77\" (UID: \"01ae1c4c-fc05-473f-ae47-078e18d88e14\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77"
Nov 03 20:29:02.807010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.806996    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmk2h\" (UniqueName: \"kubernetes.io/projected/4d4bda65-68f7-4c52-95ff-b0d7911b05f1-kube-api-access-dmk2h\") pod \"service-ca-operator-544d749c8c-mgfzm\" (UID: \"4d4bda65-68f7-4c52-95ff-b0d7911b05f1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"
Nov 03 20:29:02.807046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.807025    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lz9d\" (UniqueName: \"kubernetes.io/projected/f0adafcb-979f-4010-966f-d6f216688606-kube-api-access-4lz9d\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg\" (UID: \"f0adafcb-979f-4010-966f-d6f216688606\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"
Nov 03 20:29:02.807091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.807052    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01ae1c4c-fc05-473f-ae47-078e18d88e14-signing-cabundle\") pod \"service-ca-5fffc97bdf-mnp77\" (UID: \"01ae1c4c-fc05-473f-ae47-078e18d88e14\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77"
Nov 03 20:29:02.807244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.807225    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/764bdfb5-060a-4657-b9e0-254f1795eea8-config\") pod \"console-operator-7cd56b8dbc-6k7gv\" (UID: \"764bdfb5-060a-4657-b9e0-254f1795eea8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:02.807322 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.807295    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0adafcb-979f-4010-966f-d6f216688606-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg\" (UID: \"f0adafcb-979f-4010-966f-d6f216688606\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"
Nov 03 20:29:02.807362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.807345    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns9tf\" (UniqueName: \"kubernetes.io/projected/764bdfb5-060a-4657-b9e0-254f1795eea8-kube-api-access-ns9tf\") pod \"console-operator-7cd56b8dbc-6k7gv\" (UID: \"764bdfb5-060a-4657-b9e0-254f1795eea8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:02.807499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.807470    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d4bda65-68f7-4c52-95ff-b0d7911b05f1-config\") pod \"service-ca-operator-544d749c8c-mgfzm\" (UID: \"4d4bda65-68f7-4c52-95ff-b0d7911b05f1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"
Nov 03 20:29:02.807538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.807516    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0a72b034-13ca-4e05-babf-97728947b2ac-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6tkgl\" (UID: \"0a72b034-13ca-4e05-babf-97728947b2ac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"
Nov 03 20:29:02.807646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.807631    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0adafcb-979f-4010-966f-d6f216688606-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg\" (UID: \"f0adafcb-979f-4010-966f-d6f216688606\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"
Nov 03 20:29:02.807711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.807696    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d4bda65-68f7-4c52-95ff-b0d7911b05f1-serving-cert\") pod \"service-ca-operator-544d749c8c-mgfzm\" (UID: \"4d4bda65-68f7-4c52-95ff-b0d7911b05f1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"
Nov 03 20:29:02.807778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.807759    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 20:29:02.807833 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.807777    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/764bdfb5-060a-4657-b9e0-254f1795eea8-trusted-ca\") pod \"console-operator-7cd56b8dbc-6k7gv\" (UID: \"764bdfb5-060a-4657-b9e0-254f1795eea8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:02.807883 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.807872    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 20:29:02.807938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.807916    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:02.808103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.808086    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4d4bda65-68f7-4c52-95ff-b0d7911b05f1-config\") pod \"service-ca-operator-544d749c8c-mgfzm\" (UID: \"4d4bda65-68f7-4c52-95ff-b0d7911b05f1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"
Nov 03 20:29:02.810334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.810308    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4d4bda65-68f7-4c52-95ff-b0d7911b05f1-serving-cert\") pod \"service-ca-operator-544d749c8c-mgfzm\" (UID: \"4d4bda65-68f7-4c52-95ff-b0d7911b05f1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"
Nov 03 20:29:02.819148 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.819123    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmk2h\" (UniqueName: \"kubernetes.io/projected/4d4bda65-68f7-4c52-95ff-b0d7911b05f1-kube-api-access-dmk2h\") pod \"service-ca-operator-544d749c8c-mgfzm\" (UID: \"4d4bda65-68f7-4c52-95ff-b0d7911b05f1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"
Nov 03 20:29:02.884037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.883998    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"]
Nov 03 20:29:02.887114 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:29:02.887049    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cfde48_50fa_46b8_8d5f_645f54101110.slice/crio-1bd658e4da15b8d4e138b0f3220dad5ed27ab616189c01f037f8fb30873e0d6f WatchSource:0}: Error finding container 1bd658e4da15b8d4e138b0f3220dad5ed27ab616189c01f037f8fb30873e0d6f: Status 404 returned error can't find the container with id 1bd658e4da15b8d4e138b0f3220dad5ed27ab616189c01f037f8fb30873e0d6f
Nov 03 20:29:02.909198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909171    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/764bdfb5-060a-4657-b9e0-254f1795eea8-trusted-ca\") pod \"console-operator-7cd56b8dbc-6k7gv\" (UID: \"764bdfb5-060a-4657-b9e0-254f1795eea8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:02.909299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909211    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69kq\" (UniqueName: \"kubernetes.io/projected/0427c980-df47-4ed8-bb46-0c43c7911c4e-kube-api-access-k69kq\") pod \"cluster-samples-operator-6ddb6c8b94-jj92x\" (UID: \"0427c980-df47-4ed8-bb46-0c43c7911c4e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"
Nov 03 20:29:02.909299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909257    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5tnw\" (UniqueName: \"kubernetes.io/projected/419f6a62-6603-4ad3-b1b1-144adae9ffca-kube-api-access-w5tnw\") pod \"migrator-66df5d496b-pqq2n\" (UID: \"419f6a62-6603-4ad3-b1b1-144adae9ffca\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pqq2n"
Nov 03 20:29:02.909299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909275    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a72b034-13ca-4e05-babf-97728947b2ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6tkgl\" (UID: \"0a72b034-13ca-4e05-babf-97728947b2ac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"
Nov 03 20:29:02.909299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909298    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hs7j\" (UniqueName: \"kubernetes.io/projected/0a72b034-13ca-4e05-babf-97728947b2ac-kube-api-access-2hs7j\") pod \"cluster-monitoring-operator-884576c57-6tkgl\" (UID: \"0a72b034-13ca-4e05-babf-97728947b2ac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"
Nov 03 20:29:02.909541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909326    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01ae1c4c-fc05-473f-ae47-078e18d88e14-signing-key\") pod \"service-ca-5fffc97bdf-mnp77\" (UID: \"01ae1c4c-fc05-473f-ae47-078e18d88e14\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77"
Nov 03 20:29:02.909541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909389    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0427c980-df47-4ed8-bb46-0c43c7911c4e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jj92x\" (UID: \"0427c980-df47-4ed8-bb46-0c43c7911c4e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"
Nov 03 20:29:02.909541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909422    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/764bdfb5-060a-4657-b9e0-254f1795eea8-serving-cert\") pod \"console-operator-7cd56b8dbc-6k7gv\" (UID: \"764bdfb5-060a-4657-b9e0-254f1795eea8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:02.909541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909438    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13c92206-9485-47b9-a056-dcb8ec3588c4-tmp\") pod \"downloads-6dd789c8b7-8dg79\" (UID: \"13c92206-9485-47b9-a056-dcb8ec3588c4\") " pod="openshift-console/downloads-6dd789c8b7-8dg79"
Nov 03 20:29:02.909541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909459    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0427c980-df47-4ed8-bb46-0c43c7911c4e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jj92x\" (UID: \"0427c980-df47-4ed8-bb46-0c43c7911c4e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"
Nov 03 20:29:02.909541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909477    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbgqs\" (UniqueName: \"kubernetes.io/projected/01ae1c4c-fc05-473f-ae47-078e18d88e14-kube-api-access-wbgqs\") pod \"service-ca-5fffc97bdf-mnp77\" (UID: \"01ae1c4c-fc05-473f-ae47-078e18d88e14\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77"
Nov 03 20:29:02.909541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909515    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5h7x\" (UniqueName: \"kubernetes.io/projected/13c92206-9485-47b9-a056-dcb8ec3588c4-kube-api-access-f5h7x\") pod \"downloads-6dd789c8b7-8dg79\" (UID: \"13c92206-9485-47b9-a056-dcb8ec3588c4\") " pod="openshift-console/downloads-6dd789c8b7-8dg79"
Nov 03 20:29:02.909922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909558    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lz9d\" (UniqueName: \"kubernetes.io/projected/f0adafcb-979f-4010-966f-d6f216688606-kube-api-access-4lz9d\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg\" (UID: \"f0adafcb-979f-4010-966f-d6f216688606\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"
Nov 03 20:29:02.909922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909582    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01ae1c4c-fc05-473f-ae47-078e18d88e14-signing-cabundle\") pod \"service-ca-5fffc97bdf-mnp77\" (UID: \"01ae1c4c-fc05-473f-ae47-078e18d88e14\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77"
Nov 03 20:29:02.909922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909618    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/764bdfb5-060a-4657-b9e0-254f1795eea8-config\") pod \"console-operator-7cd56b8dbc-6k7gv\" (UID: \"764bdfb5-060a-4657-b9e0-254f1795eea8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:02.909922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909652    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0adafcb-979f-4010-966f-d6f216688606-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg\" (UID: \"f0adafcb-979f-4010-966f-d6f216688606\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"
Nov 03 20:29:02.909922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909682    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns9tf\" (UniqueName: \"kubernetes.io/projected/764bdfb5-060a-4657-b9e0-254f1795eea8-kube-api-access-ns9tf\") pod \"console-operator-7cd56b8dbc-6k7gv\" (UID: \"764bdfb5-060a-4657-b9e0-254f1795eea8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:02.909922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909746    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0a72b034-13ca-4e05-babf-97728947b2ac-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6tkgl\" (UID: \"0a72b034-13ca-4e05-babf-97728947b2ac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"
Nov 03 20:29:02.909922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.909769    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0adafcb-979f-4010-966f-d6f216688606-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg\" (UID: \"f0adafcb-979f-4010-966f-d6f216688606\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"
Nov 03 20:29:02.910282 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.910245    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0adafcb-979f-4010-966f-d6f216688606-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg\" (UID: \"f0adafcb-979f-4010-966f-d6f216688606\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"
Nov 03 20:29:02.911102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.911079    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/764bdfb5-060a-4657-b9e0-254f1795eea8-trusted-ca\") pod \"console-operator-7cd56b8dbc-6k7gv\" (UID: \"764bdfb5-060a-4657-b9e0-254f1795eea8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:02.911245 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.911224    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01ae1c4c-fc05-473f-ae47-078e18d88e14-signing-cabundle\") pod \"service-ca-5fffc97bdf-mnp77\" (UID: \"01ae1c4c-fc05-473f-ae47-078e18d88e14\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77"
Nov 03 20:29:02.911721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.911700    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/764bdfb5-060a-4657-b9e0-254f1795eea8-config\") pod \"console-operator-7cd56b8dbc-6k7gv\" (UID: \"764bdfb5-060a-4657-b9e0-254f1795eea8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:02.912776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.912748    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0a72b034-13ca-4e05-babf-97728947b2ac-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6tkgl\" (UID: \"0a72b034-13ca-4e05-babf-97728947b2ac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"
Nov 03 20:29:02.914781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.914758    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/764bdfb5-060a-4657-b9e0-254f1795eea8-serving-cert\") pod \"console-operator-7cd56b8dbc-6k7gv\" (UID: \"764bdfb5-060a-4657-b9e0-254f1795eea8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:02.914887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.914758    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0adafcb-979f-4010-966f-d6f216688606-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg\" (UID: \"f0adafcb-979f-4010-966f-d6f216688606\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"
Nov 03 20:29:02.914951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.914919    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a72b034-13ca-4e05-babf-97728947b2ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6tkgl\" (UID: \"0a72b034-13ca-4e05-babf-97728947b2ac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"
Nov 03 20:29:02.915122 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.915106    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01ae1c4c-fc05-473f-ae47-078e18d88e14-signing-key\") pod \"service-ca-5fffc97bdf-mnp77\" (UID: \"01ae1c4c-fc05-473f-ae47-078e18d88e14\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77"
Nov 03 20:29:02.924209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.924180    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lz9d\" (UniqueName: \"kubernetes.io/projected/f0adafcb-979f-4010-966f-d6f216688606-kube-api-access-4lz9d\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg\" (UID: \"f0adafcb-979f-4010-966f-d6f216688606\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"
Nov 03 20:29:02.924326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.924182    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbgqs\" (UniqueName: \"kubernetes.io/projected/01ae1c4c-fc05-473f-ae47-078e18d88e14-kube-api-access-wbgqs\") pod \"service-ca-5fffc97bdf-mnp77\" (UID: \"01ae1c4c-fc05-473f-ae47-078e18d88e14\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77"
Nov 03 20:29:02.924326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.924293    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hs7j\" (UniqueName: \"kubernetes.io/projected/0a72b034-13ca-4e05-babf-97728947b2ac-kube-api-access-2hs7j\") pod \"cluster-monitoring-operator-884576c57-6tkgl\" (UID: \"0a72b034-13ca-4e05-babf-97728947b2ac\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"
Nov 03 20:29:02.924418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.924335    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5tnw\" (UniqueName: \"kubernetes.io/projected/419f6a62-6603-4ad3-b1b1-144adae9ffca-kube-api-access-w5tnw\") pod \"migrator-66df5d496b-pqq2n\" (UID: \"419f6a62-6603-4ad3-b1b1-144adae9ffca\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pqq2n"
Nov 03 20:29:02.924913 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.924892    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns9tf\" (UniqueName: \"kubernetes.io/projected/764bdfb5-060a-4657-b9e0-254f1795eea8-kube-api-access-ns9tf\") pod \"console-operator-7cd56b8dbc-6k7gv\" (UID: \"764bdfb5-060a-4657-b9e0-254f1795eea8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:02.926856 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.926836    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"
Nov 03 20:29:02.945827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.945797    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77"
Nov 03 20:29:02.973917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.973885    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"
Nov 03 20:29:02.987054 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.987026    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"
Nov 03 20:29:02.996261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:02.996232    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:03.011825 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.011795    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69kq\" (UniqueName: \"kubernetes.io/projected/0427c980-df47-4ed8-bb46-0c43c7911c4e-kube-api-access-k69kq\") pod \"cluster-samples-operator-6ddb6c8b94-jj92x\" (UID: \"0427c980-df47-4ed8-bb46-0c43c7911c4e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"
Nov 03 20:29:03.011983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.011963    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0427c980-df47-4ed8-bb46-0c43c7911c4e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jj92x\" (UID: \"0427c980-df47-4ed8-bb46-0c43c7911c4e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"
Nov 03 20:29:03.012063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.012025    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13c92206-9485-47b9-a056-dcb8ec3588c4-tmp\") pod \"downloads-6dd789c8b7-8dg79\" (UID: \"13c92206-9485-47b9-a056-dcb8ec3588c4\") " pod="openshift-console/downloads-6dd789c8b7-8dg79"
Nov 03 20:29:03.012124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.012059    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0427c980-df47-4ed8-bb46-0c43c7911c4e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jj92x\" (UID: \"0427c980-df47-4ed8-bb46-0c43c7911c4e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"
Nov 03 20:29:03.012124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.012098    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5h7x\" (UniqueName: \"kubernetes.io/projected/13c92206-9485-47b9-a056-dcb8ec3588c4-kube-api-access-f5h7x\") pod \"downloads-6dd789c8b7-8dg79\" (UID: \"13c92206-9485-47b9-a056-dcb8ec3588c4\") " pod="openshift-console/downloads-6dd789c8b7-8dg79"
Nov 03 20:29:03.012452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.012425    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13c92206-9485-47b9-a056-dcb8ec3588c4-tmp\") pod \"downloads-6dd789c8b7-8dg79\" (UID: \"13c92206-9485-47b9-a056-dcb8ec3588c4\") " pod="openshift-console/downloads-6dd789c8b7-8dg79"
Nov 03 20:29:03.012793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.012768    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0427c980-df47-4ed8-bb46-0c43c7911c4e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-jj92x\" (UID: \"0427c980-df47-4ed8-bb46-0c43c7911c4e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"
Nov 03 20:29:03.030096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.030074    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pqq2n"
Nov 03 20:29:03.033683 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.033415    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0427c980-df47-4ed8-bb46-0c43c7911c4e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-jj92x\" (UID: \"0427c980-df47-4ed8-bb46-0c43c7911c4e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"
Nov 03 20:29:03.033806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.033735    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69kq\" (UniqueName: \"kubernetes.io/projected/0427c980-df47-4ed8-bb46-0c43c7911c4e-kube-api-access-k69kq\") pod \"cluster-samples-operator-6ddb6c8b94-jj92x\" (UID: \"0427c980-df47-4ed8-bb46-0c43c7911c4e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"
Nov 03 20:29:03.033806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.033780    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5h7x\" (UniqueName: \"kubernetes.io/projected/13c92206-9485-47b9-a056-dcb8ec3588c4-kube-api-access-f5h7x\") pod \"downloads-6dd789c8b7-8dg79\" (UID: \"13c92206-9485-47b9-a056-dcb8ec3588c4\") " pod="openshift-console/downloads-6dd789c8b7-8dg79"
Nov 03 20:29:03.042750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.042724    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-8dg79"
Nov 03 20:29:03.056062 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.056026    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"
Nov 03 20:29:03.167688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.147277    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:03.289600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.289038    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:03.459002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.458967    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:03.460067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.460037    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg"]
Nov 03 20:29:03.592929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.592885    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mnp77"]
Nov 03 20:29:03.594277 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:29:03.594250    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ae1c4c_fc05_473f_ae47_078e18d88e14.slice/crio-2a3ff8d53efc8d6020f91e68bb8c18c4eef76a4bb6667df5602cd378854a778c WatchSource:0}: Error finding container 2a3ff8d53efc8d6020f91e68bb8c18c4eef76a4bb6667df5602cd378854a778c: Status 404 returned error can't find the container with id 2a3ff8d53efc8d6020f91e68bb8c18c4eef76a4bb6667df5602cd378854a778c
Nov 03 20:29:03.605954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.605252    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm"]
Nov 03 20:29:03.612837 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.612760    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"]
Nov 03 20:29:03.616544 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:29:03.616333    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a72b034_13ca_4e05_babf_97728947b2ac.slice/crio-f0bd71566ba74620fcf301d9d62f16e797f7f3e695757b1ce57f19331d9ade87 WatchSource:0}: Error finding container f0bd71566ba74620fcf301d9d62f16e797f7f3e695757b1ce57f19331d9ade87: Status 404 returned error can't find the container with id f0bd71566ba74620fcf301d9d62f16e797f7f3e695757b1ce57f19331d9ade87
Nov 03 20:29:03.620430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.619364    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl"]
Nov 03 20:29:03.628102 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:29:03.627646    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4bda65_68f7_4c52_95ff_b0d7911b05f1.slice/crio-84a6f486b56d1bc5f20005ee4b60ff17e622352776b04d954c52ac833b73d6d8 WatchSource:0}: Error finding container 84a6f486b56d1bc5f20005ee4b60ff17e622352776b04d954c52ac833b73d6d8: Status 404 returned error can't find the container with id 84a6f486b56d1bc5f20005ee4b60ff17e622352776b04d954c52ac833b73d6d8
Nov 03 20:29:03.665562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.664417    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pqq2n"]
Nov 03 20:29:03.683985 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.683959    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x"]
Nov 03 20:29:03.700091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.700063    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-8dg79"]
Nov 03 20:29:03.701272 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:29:03.701245    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c92206_9485_47b9_a056_dcb8ec3588c4.slice/crio-e0681e6af7c662714a9e211c5a2fbe4e8adc50ace897bed1cd78755727a24307 WatchSource:0}: Error finding container e0681e6af7c662714a9e211c5a2fbe4e8adc50ace897bed1cd78755727a24307: Status 404 returned error can't find the container with id e0681e6af7c662714a9e211c5a2fbe4e8adc50ace897bed1cd78755727a24307
Nov 03 20:29:03.859106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.859060    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv" event={"ID":"764bdfb5-060a-4657-b9e0-254f1795eea8","Type":"ContainerStarted","Data":"2a45fc1626a22d7eb518cd526c8c13b470c922646f21e8d747c45660bdd47029"}
Nov 03 20:29:03.860896 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.860868    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw" event={"ID":"96cfde48-50fa-46b8-8d5f-645f54101110","Type":"ContainerStarted","Data":"f973ceb6fdaf838ad19ffa12200cc3da29836f7dd5e95d21a45ee7099e6b2646"}
Nov 03 20:29:03.861036 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.860917    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw" event={"ID":"96cfde48-50fa-46b8-8d5f-645f54101110","Type":"ContainerStarted","Data":"1bd658e4da15b8d4e138b0f3220dad5ed27ab616189c01f037f8fb30873e0d6f"}
Nov 03 20:29:03.862375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.862347    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-8dg79" event={"ID":"13c92206-9485-47b9-a056-dcb8ec3588c4","Type":"ContainerStarted","Data":"e0681e6af7c662714a9e211c5a2fbe4e8adc50ace897bed1cd78755727a24307"}
Nov 03 20:29:03.863673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.863650    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pqq2n" event={"ID":"419f6a62-6603-4ad3-b1b1-144adae9ffca","Type":"ContainerStarted","Data":"b058eb3ca457ac83743b90f85e7c4861edab72770ddf2a9f1fcc47be1f4e5cb4"}
Nov 03 20:29:03.864921 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.864892    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm" event={"ID":"4d4bda65-68f7-4c52-95ff-b0d7911b05f1","Type":"ContainerStarted","Data":"84a6f486b56d1bc5f20005ee4b60ff17e622352776b04d954c52ac833b73d6d8"}
Nov 03 20:29:03.866271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.866245    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl" event={"ID":"0a72b034-13ca-4e05-babf-97728947b2ac","Type":"ContainerStarted","Data":"f0bd71566ba74620fcf301d9d62f16e797f7f3e695757b1ce57f19331d9ade87"}
Nov 03 20:29:03.867557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.867536    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg" event={"ID":"f0adafcb-979f-4010-966f-d6f216688606","Type":"ContainerStarted","Data":"88e8f57c324ae45408ca92160221b5031ecef7e7c622b6474560f15dddea6c7a"}
Nov 03 20:29:03.868795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.868777    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77" event={"ID":"01ae1c4c-fc05-473f-ae47-078e18d88e14","Type":"ContainerStarted","Data":"2a3ff8d53efc8d6020f91e68bb8c18c4eef76a4bb6667df5602cd378854a778c"}
Nov 03 20:29:03.870106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.870066    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x" event={"ID":"0427c980-df47-4ed8-bb46-0c43c7911c4e","Type":"ContainerStarted","Data":"e307450e0347ac68e48e865c2473e2ddca746fa368847138a7c71707a4d0caf7"}
Nov 03 20:29:03.887492 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:03.887424    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw" podStartSLOduration=1.8874084230000001 podStartE2EDuration="1.887408423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:03.885431912 +0000 UTC m=+3096.040334334" watchObservedRunningTime="2026-11-03 20:29:03.887408423 +0000 UTC m=+3096.042310843"
Nov 03 20:29:04.136241 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:04.136203    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:04.912312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:04.912266    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:04.927946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:04.927776    5659 generic.go:356] "Generic (PLEG): container finished" podID="96cfde48-50fa-46b8-8d5f-645f54101110" containerID="f973ceb6fdaf838ad19ffa12200cc3da29836f7dd5e95d21a45ee7099e6b2646" exitCode=22
Nov 03 20:29:04.927946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:04.927850    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw" event={"ID":"96cfde48-50fa-46b8-8d5f-645f54101110","Type":"ContainerDied","Data":"f973ceb6fdaf838ad19ffa12200cc3da29836f7dd5e95d21a45ee7099e6b2646"}
Nov 03 20:29:04.931011 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:04.930988    5659 scope.go:122] "RemoveContainer" containerID="f973ceb6fdaf838ad19ffa12200cc3da29836f7dd5e95d21a45ee7099e6b2646"
Nov 03 20:29:05.292407 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:05.292346    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:05.835679 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:05.835579    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:06.621510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:06.620055    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:07.015501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:07.014985    5659 generic.go:356] "Generic (PLEG): container finished" podID="96cfde48-50fa-46b8-8d5f-645f54101110" containerID="05903d88a6dacfca038d957a650af03470216e0432e30ffa0cc01704d5377b53" exitCode=22
Nov 03 20:29:07.016363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:07.016316    5659 scope.go:122] "RemoveContainer" containerID="05903d88a6dacfca038d957a650af03470216e0432e30ffa0cc01704d5377b53"
Nov 03 20:29:07.016674 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:07.016438    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw" event={"ID":"96cfde48-50fa-46b8-8d5f-645f54101110","Type":"ContainerDied","Data":"05903d88a6dacfca038d957a650af03470216e0432e30ffa0cc01704d5377b53"}
Nov 03 20:29:07.016674 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:07.016637    5659 scope.go:122] "RemoveContainer" containerID="f973ceb6fdaf838ad19ffa12200cc3da29836f7dd5e95d21a45ee7099e6b2646"
Nov 03 20:29:07.016983 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:07.016954    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-lv7jw_odh-ai-gateway-infra(96cfde48-50fa-46b8-8d5f-645f54101110)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw" podUID="96cfde48-50fa-46b8-8d5f-645f54101110"
Nov 03 20:29:07.241235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:07.241128    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:07.461369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:07.461330    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:08.195583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:08.186705    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:08.295094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:08.295064    5659 scope.go:122] "RemoveContainer" containerID="05903d88a6dacfca038d957a650af03470216e0432e30ffa0cc01704d5377b53"
Nov 03 20:29:08.299228 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:08.295884    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-lv7jw_odh-ai-gateway-infra(96cfde48-50fa-46b8-8d5f-645f54101110)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw" podUID="96cfde48-50fa-46b8-8d5f-645f54101110"
Nov 03 20:29:08.394931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:08.394888    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pqq2n" event={"ID":"419f6a62-6603-4ad3-b1b1-144adae9ffca","Type":"ContainerStarted","Data":"09586284587d80c4dd1df80804fc5fde68ffbee3bd629a0637d8d2e45876bc71"}
Nov 03 20:29:08.436388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:08.436347    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:08.439009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:08.438950    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pqq2n" podStartSLOduration=2.550683363 podStartE2EDuration="6.438931814s" totalImagesPullingTime="3.888248451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:08.41492562 +0000 UTC m=+3100.569828041" watchObservedRunningTime="2026-11-03 20:29:08.438931814 +0000 UTC m=+3100.593834234"
Nov 03 20:29:09.146605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:09.146557    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:09.831530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:09.831464    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:09.978198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:09.978172    5659 scope.go:122] "RemoveContainer" containerID="8957882e014366dff43a87f9bbb26177b8623186e7bae6d9c9b760c4133c18e1"
Nov 03 20:29:10.526778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:10.526695    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:11.293571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:11.288123    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:11.493474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:11.493398    5659 generic.go:356] "Generic (PLEG): container finished" podID="57807933-a9af-459d-a489-698687afaa45" containerID="002480a9b991492dfa6d112fbabd6f27dd510f08a6d528259f6cfbd8264007ff" exitCode=22
Nov 03 20:29:11.493828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:11.493787    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b" event={"ID":"57807933-a9af-459d-a489-698687afaa45","Type":"ContainerDied","Data":"002480a9b991492dfa6d112fbabd6f27dd510f08a6d528259f6cfbd8264007ff"}
Nov 03 20:29:11.493914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:11.493860    5659 scope.go:122] "RemoveContainer" containerID="8957882e014366dff43a87f9bbb26177b8623186e7bae6d9c9b760c4133c18e1"
Nov 03 20:29:11.494720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:11.494662    5659 scope.go:122] "RemoveContainer" containerID="002480a9b991492dfa6d112fbabd6f27dd510f08a6d528259f6cfbd8264007ff"
Nov 03 20:29:11.495354 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:11.495288    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-6d22b_odh-ai-gateway-infra(57807933-a9af-459d-a489-698687afaa45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b" podUID="57807933-a9af-459d-a489-698687afaa45"
Nov 03 20:29:11.497110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:11.497065    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl" event={"ID":"0a72b034-13ca-4e05-babf-97728947b2ac","Type":"ContainerStarted","Data":"8be5a3661fe96fb6deb981859d9e343a357c361c1d35be10c16caa5ba4e004ab"}
Nov 03 20:29:11.507568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:11.507543    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x" event={"ID":"0427c980-df47-4ed8-bb46-0c43c7911c4e","Type":"ContainerStarted","Data":"cbe0eeb9a2ab03b9ab8629ce9aebf2816191f3020dfb443b45c61d1126a00040"}
Nov 03 20:29:11.507683 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:11.507588    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x" event={"ID":"0427c980-df47-4ed8-bb46-0c43c7911c4e","Type":"ContainerStarted","Data":"284ba76ce439a6ee3adf7d986d3a6357d580ccc4fd04e9a9e3dddabba2d9d341"}
Nov 03 20:29:11.543018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:11.542957    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-jj92x" podStartSLOduration=2.774553965 podStartE2EDuration="9.542934308s" totalImagesPullingTime="6.768380343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:11.535769682 +0000 UTC m=+3103.690672103" watchObservedRunningTime="2026-11-03 20:29:11.542934308 +0000 UTC m=+3103.697836735"
Nov 03 20:29:11.572779 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:11.572669    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6tkgl" podStartSLOduration=2.315401624 podStartE2EDuration="9.572650446s" totalImagesPullingTime="7.257248822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:11.564993343 +0000 UTC m=+3103.719895761" watchObservedRunningTime="2026-11-03 20:29:11.572650446 +0000 UTC m=+3103.727552866"
Nov 03 20:29:12.063505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:12.060121    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:12.529304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:12.529232    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv" event={"ID":"764bdfb5-060a-4657-b9e0-254f1795eea8","Type":"ContainerStarted","Data":"333a74401505a8757deaefa454392ced334ef17b54bda68f787946eb2d34b9be"}
Nov 03 20:29:12.529979 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:12.529949    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:12.535848 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:12.535815    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg" event={"ID":"f0adafcb-979f-4010-966f-d6f216688606","Type":"ContainerStarted","Data":"0616d8392a1b923723dc432ce29c9cc677cbdde35f2b5adf4bc2c8c43440e654"}
Nov 03 20:29:12.538753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:12.538724    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"]
Nov 03 20:29:12.577009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:12.576757    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv"
Nov 03 20:29:12.582236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:12.582171    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-6k7gv" podStartSLOduration=2.73033588 podStartE2EDuration="10.582149234s" totalImagesPullingTime="7.851813354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:12.574650688 +0000 UTC m=+3104.729553108" watchObservedRunningTime="2026-11-03 20:29:12.582149234 +0000 UTC m=+3104.737051654"
Nov 03 20:29:12.609431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:12.608595    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9n4wg" podStartSLOduration=2.087464605 podStartE2EDuration="10.608580181s" totalImagesPullingTime="8.521115576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:12.60440543 +0000 UTC m=+3104.759307850" watchObservedRunningTime="2026-11-03 20:29:12.608580181 +0000 UTC m=+3104.763482599"
Nov 03 20:29:12.746662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:12.746604    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:12.950812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:12.950653    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"
Nov 03 20:29:13.001773 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.001738    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57807933-a9af-459d-a489-698687afaa45-kube-api-access-47vzh\" (UniqueName: \"kubernetes.io/projected/57807933-a9af-459d-a489-698687afaa45-kube-api-access-47vzh\") pod \"57807933-a9af-459d-a489-698687afaa45\" (UID: \"57807933-a9af-459d-a489-698687afaa45\") "
Nov 03 20:29:13.002302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.002274    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57807933-a9af-459d-a489-698687afaa45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57807933-a9af-459d-a489-698687afaa45-maas-api-xks-ca\") pod \"57807933-a9af-459d-a489-698687afaa45\" (UID: \"57807933-a9af-459d-a489-698687afaa45\") "
Nov 03 20:29:13.002406 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.002334    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57807933-a9af-459d-a489-698687afaa45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57807933-a9af-459d-a489-698687afaa45-maas-api-openshift-ca\") pod \"57807933-a9af-459d-a489-698687afaa45\" (UID: \"57807933-a9af-459d-a489-698687afaa45\") "
Nov 03 20:29:13.003096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.003068    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57807933-a9af-459d-a489-698687afaa45-maas-api-openshift-ca" pod "57807933-a9af-459d-a489-698687afaa45" (UID: "57807933-a9af-459d-a489-698687afaa45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:13.004780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.004753    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57807933-a9af-459d-a489-698687afaa45-kube-api-access-47vzh" pod "57807933-a9af-459d-a489-698687afaa45" (UID: "57807933-a9af-459d-a489-698687afaa45"). InnerVolumeSpecName "kube-api-access-47vzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:13.005718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.005691    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57807933-a9af-459d-a489-698687afaa45-maas-api-xks-ca" pod "57807933-a9af-459d-a489-698687afaa45" (UID: "57807933-a9af-459d-a489-698687afaa45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:13.106695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.105202    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57807933-a9af-459d-a489-698687afaa45-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:13.106695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.105235    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57807933-a9af-459d-a489-698687afaa45-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:13.106695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.105251    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47vzh\" (UniqueName: \"kubernetes.io/projected/57807933-a9af-459d-a489-698687afaa45-kube-api-access-47vzh\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:13.518356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.518314    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:13.637010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.636943    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm" event={"ID":"4d4bda65-68f7-4c52-95ff-b0d7911b05f1","Type":"ContainerStarted","Data":"02406d6f7368944102b2f0864a2840c9c168414d280e97c6f06939a280636ed1"}
Nov 03 20:29:13.644560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.643071    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77" event={"ID":"01ae1c4c-fc05-473f-ae47-078e18d88e14","Type":"ContainerStarted","Data":"0bad3ad6e23ca3733b141961b041a33041f45f0b67aed515e72928cf370a8402"}
Nov 03 20:29:13.650082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.650050    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"
Nov 03 20:29:13.656880 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.650285    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b" event={"ID":"57807933-a9af-459d-a489-698687afaa45","Type":"ContainerDied","Data":"9fe1635c612b3f8e8b4564f57d43ed48531358fbeeaa27a9312fd3884874d74e"}
Nov 03 20:29:13.656880 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.656867    5659 scope.go:122] "RemoveContainer" containerID="002480a9b991492dfa6d112fbabd6f27dd510f08a6d528259f6cfbd8264007ff"
Nov 03 20:29:13.667452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.665947    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgfzm" podStartSLOduration=2.340319293 podStartE2EDuration="11.665929589s" totalImagesPullingTime="9.325610296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:13.657881226 +0000 UTC m=+3105.812783642" watchObservedRunningTime="2026-11-03 20:29:13.665929589 +0000 UTC m=+3105.820832008"
Nov 03 20:29:13.695804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.695743    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-mnp77" podStartSLOduration=2.714818287 podStartE2EDuration="11.695722475s" totalImagesPullingTime="8.980904188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:13.688887233 +0000 UTC m=+3105.843789649" watchObservedRunningTime="2026-11-03 20:29:13.695722475 +0000 UTC m=+3105.850624895"
Nov 03 20:29:13.717596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.717563    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"]
Nov 03 20:29:13.726286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.726241    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6d22b"]
Nov 03 20:29:13.979169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:13.979134    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57807933-a9af-459d-a489-698687afaa45" path="/var/lib/kubelet/pods/57807933-a9af-459d-a489-698687afaa45/volumes"
Nov 03 20:29:14.347070 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:14.346970    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:15.001197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:15.001165    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:15.223497 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:15.223437    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:29:15.802505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:15.796049    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:16.523710 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:16.523601    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:17.193020 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:17.192982    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:17.935240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:17.935178    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:18.626887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:18.626847    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:19.368540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:19.368500    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:20.164085 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:20.163986    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:20.930692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:20.930659    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:20.967420 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:20.967370    5659 scope.go:122] "RemoveContainer" containerID="05903d88a6dacfca038d957a650af03470216e0432e30ffa0cc01704d5377b53"
Nov 03 20:29:21.633070 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:21.632989    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:21.791843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:21.791808    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw" event={"ID":"96cfde48-50fa-46b8-8d5f-645f54101110","Type":"ContainerStarted","Data":"40ffd739a38d54924404dd70eecb9baf3be2915872a7028a6497b010aa641320"}
Nov 03 20:29:22.308941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.308901    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:22.470553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.470516    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"]
Nov 03 20:29:22.471545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.471521    5659 cpu_manager.go:414] "containerMap: removing container" podUID="57807933-a9af-459d-a489-698687afaa45" containerName="revoke-keys"
Nov 03 20:29:22.471545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.471543    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57807933-a9af-459d-a489-698687afaa45" containerName="revoke-keys"
Nov 03 20:29:22.471760 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.471651    5659 cpu_manager.go:414] "containerMap: removing container" podUID="57807933-a9af-459d-a489-698687afaa45" containerName="revoke-keys"
Nov 03 20:29:22.471760 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.471661    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57807933-a9af-459d-a489-698687afaa45" containerName="revoke-keys"
Nov 03 20:29:22.471760 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.471671    5659 cpu_manager.go:414] "containerMap: removing container" podUID="57807933-a9af-459d-a489-698687afaa45" containerName="revoke-keys"
Nov 03 20:29:22.471760 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.471678    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57807933-a9af-459d-a489-698687afaa45" containerName="revoke-keys"
Nov 03 20:29:22.472037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.471875    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="57807933-a9af-459d-a489-698687afaa45" containerName="revoke-keys"
Nov 03 20:29:22.472037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.471892    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="57807933-a9af-459d-a489-698687afaa45" containerName="revoke-keys"
Nov 03 20:29:22.472037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.471906    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="57807933-a9af-459d-a489-698687afaa45" containerName="revoke-keys"
Nov 03 20:29:22.476721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.476696    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"
Nov 03 20:29:22.484021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.483991    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"]
Nov 03 20:29:22.574898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.574860    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-hkmcv\" (UID: \"d1c367cc-759a-41c1-8640-00cf7b3b543d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"
Nov 03 20:29:22.575096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.574927    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr88q\" (UniqueName: \"kubernetes.io/projected/d1c367cc-759a-41c1-8640-00cf7b3b543d-kube-api-access-lr88q\") pod \"maas-api-revoke-sub-5bac6f1e-hkmcv\" (UID: \"d1c367cc-759a-41c1-8640-00cf7b3b543d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"
Nov 03 20:29:22.575096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.575013    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-hkmcv\" (UID: \"d1c367cc-759a-41c1-8640-00cf7b3b543d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"
Nov 03 20:29:22.676272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.676235    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr88q\" (UniqueName: \"kubernetes.io/projected/d1c367cc-759a-41c1-8640-00cf7b3b543d-kube-api-access-lr88q\") pod \"maas-api-revoke-sub-5bac6f1e-hkmcv\" (UID: \"d1c367cc-759a-41c1-8640-00cf7b3b543d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"
Nov 03 20:29:22.676455 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.676390    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-hkmcv\" (UID: \"d1c367cc-759a-41c1-8640-00cf7b3b543d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"
Nov 03 20:29:22.676745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.676717    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-hkmcv\" (UID: \"d1c367cc-759a-41c1-8640-00cf7b3b543d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"
Nov 03 20:29:22.677811 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.677779    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-hkmcv\" (UID: \"d1c367cc-759a-41c1-8640-00cf7b3b543d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"
Nov 03 20:29:22.679860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.679827    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-hkmcv\" (UID: \"d1c367cc-759a-41c1-8640-00cf7b3b543d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"
Nov 03 20:29:22.689748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.689728    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr88q\" (UniqueName: \"kubernetes.io/projected/d1c367cc-759a-41c1-8640-00cf7b3b543d-kube-api-access-lr88q\") pod \"maas-api-revoke-sub-5bac6f1e-hkmcv\" (UID: \"d1c367cc-759a-41c1-8640-00cf7b3b543d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"
Nov 03 20:29:22.796770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.796686    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"
Nov 03 20:29:22.806753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.806719    5659 generic.go:356] "Generic (PLEG): container finished" podID="96cfde48-50fa-46b8-8d5f-645f54101110" containerID="40ffd739a38d54924404dd70eecb9baf3be2915872a7028a6497b010aa641320" exitCode=22
Nov 03 20:29:22.806935 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.806797    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw" event={"ID":"96cfde48-50fa-46b8-8d5f-645f54101110","Type":"ContainerDied","Data":"40ffd739a38d54924404dd70eecb9baf3be2915872a7028a6497b010aa641320"}
Nov 03 20:29:22.806935 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.806880    5659 scope.go:122] "RemoveContainer" containerID="05903d88a6dacfca038d957a650af03470216e0432e30ffa0cc01704d5377b53"
Nov 03 20:29:22.807435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.807405    5659 scope.go:122] "RemoveContainer" containerID="40ffd739a38d54924404dd70eecb9baf3be2915872a7028a6497b010aa641320"
Nov 03 20:29:22.808180 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:22.807900    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-lv7jw_odh-ai-gateway-infra(96cfde48-50fa-46b8-8d5f-645f54101110)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw" podUID="96cfde48-50fa-46b8-8d5f-645f54101110"
Nov 03 20:29:22.827188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:22.827158    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"]
Nov 03 20:29:23.121649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:23.121621    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"]
Nov 03 20:29:23.122035 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:29:23.121998    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c367cc_759a_41c1_8640_00cf7b3b543d.slice/crio-17ac3bca5244716ee04b660b15ef7e0405ad51a789e716d56055d4d115058343 WatchSource:0}: Error finding container 17ac3bca5244716ee04b660b15ef7e0405ad51a789e716d56055d4d115058343: Status 404 returned error can't find the container with id 17ac3bca5244716ee04b660b15ef7e0405ad51a789e716d56055d4d115058343
Nov 03 20:29:23.159409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:23.156293    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:23.820964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:23.820925    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv" event={"ID":"d1c367cc-759a-41c1-8640-00cf7b3b543d","Type":"ContainerStarted","Data":"a3249e76bfc44051a0f30e9aa6eff23ae5a05dff26bc319fe67239a9c888ef7d"}
Nov 03 20:29:23.821423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:23.820993    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv" event={"ID":"d1c367cc-759a-41c1-8640-00cf7b3b543d","Type":"ContainerStarted","Data":"17ac3bca5244716ee04b660b15ef7e0405ad51a789e716d56055d4d115058343"}
Nov 03 20:29:23.847043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:23.846996    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv" podStartSLOduration=1.8469824209999999 podStartE2EDuration="1.846982421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:23.842793812 +0000 UTC m=+3115.997696232" watchObservedRunningTime="2026-11-03 20:29:23.846982421 +0000 UTC m=+3116.001884840"
Nov 03 20:29:23.938593 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:23.938553    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:24.106621 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.106586    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"
Nov 03 20:29:24.245445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.243907    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96cfde48-50fa-46b8-8d5f-645f54101110-kube-api-access-hk9bw\" (UniqueName: \"kubernetes.io/projected/96cfde48-50fa-46b8-8d5f-645f54101110-kube-api-access-hk9bw\") pod \"96cfde48-50fa-46b8-8d5f-645f54101110\" (UID: \"96cfde48-50fa-46b8-8d5f-645f54101110\") "
Nov 03 20:29:24.245445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.244199    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-openshift-ca\") pod \"96cfde48-50fa-46b8-8d5f-645f54101110\" (UID: \"96cfde48-50fa-46b8-8d5f-645f54101110\") "
Nov 03 20:29:24.245445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.244323    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-xks-ca\") pod \"96cfde48-50fa-46b8-8d5f-645f54101110\" (UID: \"96cfde48-50fa-46b8-8d5f-645f54101110\") "
Nov 03 20:29:24.246133 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.246107    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-openshift-ca" pod "96cfde48-50fa-46b8-8d5f-645f54101110" (UID: "96cfde48-50fa-46b8-8d5f-645f54101110"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:24.248234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.248206    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-xks-ca" pod "96cfde48-50fa-46b8-8d5f-645f54101110" (UID: "96cfde48-50fa-46b8-8d5f-645f54101110"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:24.248758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.248731    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cfde48-50fa-46b8-8d5f-645f54101110-kube-api-access-hk9bw" pod "96cfde48-50fa-46b8-8d5f-645f54101110" (UID: "96cfde48-50fa-46b8-8d5f-645f54101110"). InnerVolumeSpecName "kube-api-access-hk9bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:24.347437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.347402    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:24.347628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.347442    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96cfde48-50fa-46b8-8d5f-645f54101110-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:24.347628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.347458    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk9bw\" (UniqueName: \"kubernetes.io/projected/96cfde48-50fa-46b8-8d5f-645f54101110-kube-api-access-hk9bw\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:24.725344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.725308    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:24.837527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.836085    5659 generic.go:356] "Generic (PLEG): container finished" podID="d1c367cc-759a-41c1-8640-00cf7b3b543d" containerID="a3249e76bfc44051a0f30e9aa6eff23ae5a05dff26bc319fe67239a9c888ef7d" exitCode=22
Nov 03 20:29:24.837527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.836187    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv" event={"ID":"d1c367cc-759a-41c1-8640-00cf7b3b543d","Type":"ContainerDied","Data":"a3249e76bfc44051a0f30e9aa6eff23ae5a05dff26bc319fe67239a9c888ef7d"}
Nov 03 20:29:24.837527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.836712    5659 scope.go:122] "RemoveContainer" containerID="a3249e76bfc44051a0f30e9aa6eff23ae5a05dff26bc319fe67239a9c888ef7d"
Nov 03 20:29:24.839916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.839896    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"
Nov 03 20:29:24.839990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.839893    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw" event={"ID":"96cfde48-50fa-46b8-8d5f-645f54101110","Type":"ContainerDied","Data":"1bd658e4da15b8d4e138b0f3220dad5ed27ab616189c01f037f8fb30873e0d6f"}
Nov 03 20:29:24.839990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.839965    5659 scope.go:122] "RemoveContainer" containerID="40ffd739a38d54924404dd70eecb9baf3be2915872a7028a6497b010aa641320"
Nov 03 20:29:24.876205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.876140    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"]
Nov 03 20:29:24.881810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:24.881783    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-lv7jw"]
Nov 03 20:29:25.450017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:25.449940    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:25.854395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:25.854352    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv" event={"ID":"d1c367cc-759a-41c1-8640-00cf7b3b543d","Type":"ContainerStarted","Data":"21f0193114faf7c12e3a871e007ac01ab20c1f96a78c946668b8628901007613"}
Nov 03 20:29:25.974860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:25.974830    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96cfde48-50fa-46b8-8d5f-645f54101110" path="/var/lib/kubelet/pods/96cfde48-50fa-46b8-8d5f-645f54101110/volumes"
Nov 03 20:29:26.125781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:26.125693    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:26.854037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:26.853992    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:26.964206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:26.964168    5659 generic.go:356] "Generic (PLEG): container finished" podID="d1c367cc-759a-41c1-8640-00cf7b3b543d" containerID="21f0193114faf7c12e3a871e007ac01ab20c1f96a78c946668b8628901007613" exitCode=22
Nov 03 20:29:26.964752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:26.964519    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv" event={"ID":"d1c367cc-759a-41c1-8640-00cf7b3b543d","Type":"ContainerDied","Data":"21f0193114faf7c12e3a871e007ac01ab20c1f96a78c946668b8628901007613"}
Nov 03 20:29:26.964752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:26.964572    5659 scope.go:122] "RemoveContainer" containerID="a3249e76bfc44051a0f30e9aa6eff23ae5a05dff26bc319fe67239a9c888ef7d"
Nov 03 20:29:26.964971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:26.964951    5659 scope.go:122] "RemoveContainer" containerID="21f0193114faf7c12e3a871e007ac01ab20c1f96a78c946668b8628901007613"
Nov 03 20:29:26.965361 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:26.965334    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-hkmcv_odh-ai-gateway-infra(d1c367cc-759a-41c1-8640-00cf7b3b543d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv" podUID="d1c367cc-759a-41c1-8640-00cf7b3b543d"
Nov 03 20:29:27.541215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:27.541177    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:27.978823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:27.978793    5659 scope.go:122] "RemoveContainer" containerID="21f0193114faf7c12e3a871e007ac01ab20c1f96a78c946668b8628901007613"
Nov 03 20:29:27.979317 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:27.979209    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-hkmcv_odh-ai-gateway-infra(d1c367cc-759a-41c1-8640-00cf7b3b543d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv" podUID="d1c367cc-759a-41c1-8640-00cf7b3b543d"
Nov 03 20:29:28.199554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:28.199476    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:28.940048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:28.939961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:29.630924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:29.630885    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:30.300034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:30.299994    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:31.036851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.036815    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:31.489022 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.488988    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"]
Nov 03 20:29:31.490241 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.490219    5659 cpu_manager.go:414] "containerMap: removing container" podUID="96cfde48-50fa-46b8-8d5f-645f54101110" containerName="revoke-keys"
Nov 03 20:29:31.490303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.490247    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96cfde48-50fa-46b8-8d5f-645f54101110" containerName="revoke-keys"
Nov 03 20:29:31.490303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.490264    5659 cpu_manager.go:414] "containerMap: removing container" podUID="96cfde48-50fa-46b8-8d5f-645f54101110" containerName="revoke-keys"
Nov 03 20:29:31.490303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.490272    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96cfde48-50fa-46b8-8d5f-645f54101110" containerName="revoke-keys"
Nov 03 20:29:31.490303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.490284    5659 cpu_manager.go:414] "containerMap: removing container" podUID="96cfde48-50fa-46b8-8d5f-645f54101110" containerName="revoke-keys"
Nov 03 20:29:31.490303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.490292    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96cfde48-50fa-46b8-8d5f-645f54101110" containerName="revoke-keys"
Nov 03 20:29:31.490581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.490568    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="96cfde48-50fa-46b8-8d5f-645f54101110" containerName="revoke-keys"
Nov 03 20:29:31.490618 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.490586    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="96cfde48-50fa-46b8-8d5f-645f54101110" containerName="revoke-keys"
Nov 03 20:29:31.490618 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.490596    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="96cfde48-50fa-46b8-8d5f-645f54101110" containerName="revoke-keys"
Nov 03 20:29:31.517307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.517265    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"]
Nov 03 20:29:31.517561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.517414    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"
Nov 03 20:29:31.678801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.678707    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znql2\" (UniqueName: \"kubernetes.io/projected/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-kube-api-access-znql2\") pod \"maas-api-revoke-sub-fb827799-rk2jm\" (UID: \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"
Nov 03 20:29:31.678981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.678932    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-rk2jm\" (UID: \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"
Nov 03 20:29:31.679119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.679101    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-rk2jm\" (UID: \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"
Nov 03 20:29:31.752719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.752687    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:31.785745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.781822    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-rk2jm\" (UID: \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"
Nov 03 20:29:31.785745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.782074    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znql2\" (UniqueName: \"kubernetes.io/projected/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-kube-api-access-znql2\") pod \"maas-api-revoke-sub-fb827799-rk2jm\" (UID: \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"
Nov 03 20:29:31.785745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.782230    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-rk2jm\" (UID: \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"
Nov 03 20:29:31.785745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.783171    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-rk2jm\" (UID: \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"
Nov 03 20:29:31.820742 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.820390    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-rk2jm\" (UID: \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"
Nov 03 20:29:31.847591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.840182    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znql2\" (UniqueName: \"kubernetes.io/projected/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-kube-api-access-znql2\") pod \"maas-api-revoke-sub-fb827799-rk2jm\" (UID: \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"
Nov 03 20:29:31.851471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:31.851298    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"
Nov 03 20:29:32.095998 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.095617    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"]
Nov 03 20:29:32.124574 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.124530    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"]
Nov 03 20:29:32.124760 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.124642    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"
Nov 03 20:29:32.209116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.209083    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djdjb\" (UniqueName: \"kubernetes.io/projected/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-kube-api-access-djdjb\") pod \"maas-api-revoke-sub-abad4031-dhpvf\" (UID: \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"
Nov 03 20:29:32.209555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.209525    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-dhpvf\" (UID: \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"
Nov 03 20:29:32.209705 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.209679    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-dhpvf\" (UID: \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"
Nov 03 20:29:32.211276 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:29:32.211240    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9deb2b_ae86_44e4_b8b2_9cba6cdb77d7.slice/crio-edfba1612b7f2409d8bb676b2c352ac24e352a5719e66ef0257daf424f63577a WatchSource:0}: Error finding container edfba1612b7f2409d8bb676b2c352ac24e352a5719e66ef0257daf424f63577a: Status 404 returned error can't find the container with id edfba1612b7f2409d8bb676b2c352ac24e352a5719e66ef0257daf424f63577a
Nov 03 20:29:32.211392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.211340    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"]
Nov 03 20:29:32.311816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.311776    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-dhpvf\" (UID: \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"
Nov 03 20:29:32.312009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.311896    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-dhpvf\" (UID: \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"
Nov 03 20:29:32.312009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.311956    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdjb\" (UniqueName: \"kubernetes.io/projected/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-kube-api-access-djdjb\") pod \"maas-api-revoke-sub-abad4031-dhpvf\" (UID: \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"
Nov 03 20:29:32.312570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.312545    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-dhpvf\" (UID: \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"
Nov 03 20:29:32.314775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.314749    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-dhpvf\" (UID: \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"
Nov 03 20:29:32.326075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.326031    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdjb\" (UniqueName: \"kubernetes.io/projected/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-kube-api-access-djdjb\") pod \"maas-api-revoke-sub-abad4031-dhpvf\" (UID: \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"
Nov 03 20:29:32.454230 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.454193    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"
Nov 03 20:29:32.649125 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.649085    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:32.898535 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:32.888605    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"]
Nov 03 20:29:33.070535 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:33.070449    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm" event={"ID":"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7","Type":"ContainerStarted","Data":"773e80c17a15e54dbb2e0bf1708e1a6ca58a49cc2dd5634de6af6d7b5724a6ce"}
Nov 03 20:29:33.070863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:33.070824    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm" event={"ID":"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7","Type":"ContainerStarted","Data":"edfba1612b7f2409d8bb676b2c352ac24e352a5719e66ef0257daf424f63577a"}
Nov 03 20:29:33.073162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:33.073022    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf" event={"ID":"b9f68a13-f72d-4ecb-a2a1-3263747ab43d","Type":"ContainerStarted","Data":"5c2bbf68bb956c9dd5fedb68ea35b4d6f0937ac73b188754628fe816392dda06"}
Nov 03 20:29:33.075301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:33.075271    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-8dg79" event={"ID":"13c92206-9485-47b9-a056-dcb8ec3588c4","Type":"ContainerStarted","Data":"f6ddb24c40bc35126103ed1e9575a0fac5e7333ab674bdfeb4ca31ada13e4c8e"}
Nov 03 20:29:33.075757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:33.075730    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-8dg79"
Nov 03 20:29:33.094756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:33.094727    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-8dg79"
Nov 03 20:29:33.094976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:33.094930    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm" podStartSLOduration=2.094915743 podStartE2EDuration="2.094915743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:33.094725364 +0000 UTC m=+3125.249627784" watchObservedRunningTime="2026-11-03 20:29:33.094915743 +0000 UTC m=+3125.249818162"
Nov 03 20:29:33.116423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:33.116373    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-8dg79" podStartSLOduration=2.983936069 podStartE2EDuration="31.116360037s" totalImagesPullingTime="28.132423968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:33.11377889 +0000 UTC m=+3125.268681310" watchObservedRunningTime="2026-11-03 20:29:33.116360037 +0000 UTC m=+3125.271262457"
Nov 03 20:29:33.326258 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:33.326222    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:34.024109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:34.024070    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:34.135050 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:34.104148    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf" event={"ID":"b9f68a13-f72d-4ecb-a2a1-3263747ab43d","Type":"ContainerStarted","Data":"006b13839d446cc1c4a1869762a5bae7fe27cdaf9adede2b933fb3c9c27f870a"}
Nov 03 20:29:34.140510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:34.139790    5659 generic.go:356] "Generic (PLEG): container finished" podID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" containerID="773e80c17a15e54dbb2e0bf1708e1a6ca58a49cc2dd5634de6af6d7b5724a6ce" exitCode=7
Nov 03 20:29:34.142976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:34.141303    5659 scope.go:122] "RemoveContainer" containerID="773e80c17a15e54dbb2e0bf1708e1a6ca58a49cc2dd5634de6af6d7b5724a6ce"
Nov 03 20:29:34.159034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:34.150521    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm" event={"ID":"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7","Type":"ContainerDied","Data":"773e80c17a15e54dbb2e0bf1708e1a6ca58a49cc2dd5634de6af6d7b5724a6ce"}
Nov 03 20:29:34.167028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:34.166843    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf" podStartSLOduration=2.166823648 podStartE2EDuration="2.166823648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:34.164733443 +0000 UTC m=+3126.319635845" watchObservedRunningTime="2026-11-03 20:29:34.166823648 +0000 UTC m=+3126.321726069"
Nov 03 20:29:34.917940 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:34.917904    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:35.155882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:35.155785    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm" event={"ID":"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7","Type":"ContainerStarted","Data":"a4df89a64c2e5938735144b9bbb1875e2547e7eac136bdbe68d7c82f327b5b3f"}
Nov 03 20:29:35.158382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:35.158329    5659 generic.go:356] "Generic (PLEG): container finished" podID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d" containerID="006b13839d446cc1c4a1869762a5bae7fe27cdaf9adede2b933fb3c9c27f870a" exitCode=22
Nov 03 20:29:35.158382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:35.158364    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf" event={"ID":"b9f68a13-f72d-4ecb-a2a1-3263747ab43d","Type":"ContainerDied","Data":"006b13839d446cc1c4a1869762a5bae7fe27cdaf9adede2b933fb3c9c27f870a"}
Nov 03 20:29:35.158826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:35.158804    5659 scope.go:122] "RemoveContainer" containerID="006b13839d446cc1c4a1869762a5bae7fe27cdaf9adede2b933fb3c9c27f870a"
Nov 03 20:29:35.694600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:35.685474    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:36.172362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:36.172315    5659 generic.go:356] "Generic (PLEG): container finished" podID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" containerID="a4df89a64c2e5938735144b9bbb1875e2547e7eac136bdbe68d7c82f327b5b3f" exitCode=7
Nov 03 20:29:36.172871 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:36.172465    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm" event={"ID":"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7","Type":"ContainerDied","Data":"a4df89a64c2e5938735144b9bbb1875e2547e7eac136bdbe68d7c82f327b5b3f"}
Nov 03 20:29:36.172871 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:36.172537    5659 scope.go:122] "RemoveContainer" containerID="773e80c17a15e54dbb2e0bf1708e1a6ca58a49cc2dd5634de6af6d7b5724a6ce"
Nov 03 20:29:36.173122 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:36.173102    5659 scope.go:122] "RemoveContainer" containerID="a4df89a64c2e5938735144b9bbb1875e2547e7eac136bdbe68d7c82f327b5b3f"
Nov 03 20:29:36.173649 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:36.173620    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-rk2jm_odh-ai-gateway-infra(bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm" podUID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7"
Nov 03 20:29:36.179683 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:36.178736    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf" event={"ID":"b9f68a13-f72d-4ecb-a2a1-3263747ab43d","Type":"ContainerStarted","Data":"a1ddc01153408448ef96f9e185de7f758a2d12bfcc8ee7847016e975f9b1da33"}
Nov 03 20:29:36.409180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:36.409109    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:37.226794 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:37.226761    5659 scope.go:122] "RemoveContainer" containerID="a4df89a64c2e5938735144b9bbb1875e2547e7eac136bdbe68d7c82f327b5b3f"
Nov 03 20:29:37.236299 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:37.235961    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-rk2jm_odh-ai-gateway-infra(bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm" podUID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7"
Nov 03 20:29:37.236968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:37.234797    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf" event={"ID":"b9f68a13-f72d-4ecb-a2a1-3263747ab43d","Type":"ContainerDied","Data":"a1ddc01153408448ef96f9e185de7f758a2d12bfcc8ee7847016e975f9b1da33"}
Nov 03 20:29:37.238565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:37.234766    5659 generic.go:356] "Generic (PLEG): container finished" podID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d" containerID="a1ddc01153408448ef96f9e185de7f758a2d12bfcc8ee7847016e975f9b1da33" exitCode=22
Nov 03 20:29:37.239049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:37.236563    5659 scope.go:122] "RemoveContainer" containerID="a1ddc01153408448ef96f9e185de7f758a2d12bfcc8ee7847016e975f9b1da33"
Nov 03 20:29:37.239756 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:37.239727    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-dhpvf_odh-ai-gateway-infra(b9f68a13-f72d-4ecb-a2a1-3263747ab43d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf" podUID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d"
Nov 03 20:29:37.239965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:37.238210    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:37.240131 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:37.239907    5659 scope.go:122] "RemoveContainer" containerID="006b13839d446cc1c4a1869762a5bae7fe27cdaf9adede2b933fb3c9c27f870a"
Nov 03 20:29:37.942517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:37.939912    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:38.255086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:38.255049    5659 scope.go:122] "RemoveContainer" containerID="a1ddc01153408448ef96f9e185de7f758a2d12bfcc8ee7847016e975f9b1da33"
Nov 03 20:29:38.255761 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:38.255607    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-dhpvf_odh-ai-gateway-infra(b9f68a13-f72d-4ecb-a2a1-3263747ab43d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf" podUID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d"
Nov 03 20:29:38.634503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:38.634400    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:39.502323 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:39.496009    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:39.731084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:39.731012    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:39.975509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:39.969032    5659 scope.go:122] "RemoveContainer" containerID="21f0193114faf7c12e3a871e007ac01ab20c1f96a78c946668b8628901007613"
Nov 03 20:29:40.111061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:40.111025    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:40.250422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:40.250236    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:40.896234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:40.896196    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:41.369653 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:41.369615    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv" event={"ID":"d1c367cc-759a-41c1-8640-00cf7b3b543d","Type":"ContainerStarted","Data":"30ecb7378404c9c96d97588a342041aa11dcda2ec1b215e5c4b6292b52740f5d"}
Nov 03 20:29:41.404079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:41.403891    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:41.535291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:41.535247    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:41.699622 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:41.699536    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:42.382643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:42.382607    5659 generic.go:356] "Generic (PLEG): container finished" podID="d1c367cc-759a-41c1-8640-00cf7b3b543d" containerID="30ecb7378404c9c96d97588a342041aa11dcda2ec1b215e5c4b6292b52740f5d" exitCode=22
Nov 03 20:29:42.383112 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:42.382678    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv" event={"ID":"d1c367cc-759a-41c1-8640-00cf7b3b543d","Type":"ContainerDied","Data":"30ecb7378404c9c96d97588a342041aa11dcda2ec1b215e5c4b6292b52740f5d"}
Nov 03 20:29:42.383112 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:42.382743    5659 scope.go:122] "RemoveContainer" containerID="21f0193114faf7c12e3a871e007ac01ab20c1f96a78c946668b8628901007613"
Nov 03 20:29:42.383268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:42.383209    5659 scope.go:122] "RemoveContainer" containerID="30ecb7378404c9c96d97588a342041aa11dcda2ec1b215e5c4b6292b52740f5d"
Nov 03 20:29:42.383631 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:42.383608    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-hkmcv_odh-ai-gateway-infra(d1c367cc-759a-41c1-8640-00cf7b3b543d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv" podUID="d1c367cc-759a-41c1-8640-00cf7b3b543d"
Nov 03 20:29:42.402968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:42.402924    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"]
Nov 03 20:29:42.425861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:42.425827    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:43.106808 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:43.106775    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:43.625042 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:43.625017    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"
Nov 03 20:29:43.789594 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:43.789559    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c367cc-759a-41c1-8640-00cf7b3b543d-kube-api-access-lr88q\" (UniqueName: \"kubernetes.io/projected/d1c367cc-759a-41c1-8640-00cf7b3b543d-kube-api-access-lr88q\") pod \"d1c367cc-759a-41c1-8640-00cf7b3b543d\" (UID: \"d1c367cc-759a-41c1-8640-00cf7b3b543d\") "
Nov 03 20:29:43.789814 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:43.789738    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-openshift-ca\") pod \"d1c367cc-759a-41c1-8640-00cf7b3b543d\" (UID: \"d1c367cc-759a-41c1-8640-00cf7b3b543d\") "
Nov 03 20:29:43.789814 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:43.789771    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-xks-ca\") pod \"d1c367cc-759a-41c1-8640-00cf7b3b543d\" (UID: \"d1c367cc-759a-41c1-8640-00cf7b3b543d\") "
Nov 03 20:29:43.790164 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:43.790141    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-openshift-ca" pod "d1c367cc-759a-41c1-8640-00cf7b3b543d" (UID: "d1c367cc-759a-41c1-8640-00cf7b3b543d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:43.790566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:43.790548    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:43.798655 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:43.798628    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-xks-ca" pod "d1c367cc-759a-41c1-8640-00cf7b3b543d" (UID: "d1c367cc-759a-41c1-8640-00cf7b3b543d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:43.798783 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:43.798742    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c367cc-759a-41c1-8640-00cf7b3b543d-kube-api-access-lr88q" pod "d1c367cc-759a-41c1-8640-00cf7b3b543d" (UID: "d1c367cc-759a-41c1-8640-00cf7b3b543d"). InnerVolumeSpecName "kube-api-access-lr88q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:43.848929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:43.848866    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:43.907398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:43.899542    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c367cc-759a-41c1-8640-00cf7b3b543d-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:43.907727 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:43.907692    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr88q\" (UniqueName: \"kubernetes.io/projected/d1c367cc-759a-41c1-8640-00cf7b3b543d-kube-api-access-lr88q\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:44.417764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:44.417729    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"
Nov 03 20:29:44.417956 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:44.417726    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv" event={"ID":"d1c367cc-759a-41c1-8640-00cf7b3b543d","Type":"ContainerDied","Data":"17ac3bca5244716ee04b660b15ef7e0405ad51a789e716d56055d4d115058343"}
Nov 03 20:29:44.417956 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:44.417892    5659 scope.go:122] "RemoveContainer" containerID="30ecb7378404c9c96d97588a342041aa11dcda2ec1b215e5c4b6292b52740f5d"
Nov 03 20:29:44.445659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:44.445619    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"]
Nov 03 20:29:44.455439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:44.455407    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-hkmcv"]
Nov 03 20:29:44.641300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:44.641263    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:45.220055 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:45.220021    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:29:45.299670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:45.299638    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:45.978265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:45.978227    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c367cc-759a-41c1-8640-00cf7b3b543d" path="/var/lib/kubelet/pods/d1c367cc-759a-41c1-8640-00cf7b3b543d/volumes"
Nov 03 20:29:46.230971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:46.230852    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:46.926924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:46.926876    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:47.245034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:47.244961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:47.385422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:47.385389    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:47.588996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:47.588924    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:48.328316 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:48.328287    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:48.961609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:48.961576    5659 scope.go:122] "RemoveContainer" containerID="a4df89a64c2e5938735144b9bbb1875e2547e7eac136bdbe68d7c82f327b5b3f"
Nov 03 20:29:49.189816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:49.189783    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:49.504949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:49.504917    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm" event={"ID":"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7","Type":"ContainerStarted","Data":"fae55daa3b63d839f1ed39b4cdcae71d8ec149cd380b3f3a5f1defc9b0926d34"}
Nov 03 20:29:49.907335 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:49.907301    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:50.518731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:50.518693    5659 generic.go:356] "Generic (PLEG): container finished" podID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" containerID="fae55daa3b63d839f1ed39b4cdcae71d8ec149cd380b3f3a5f1defc9b0926d34" exitCode=7
Nov 03 20:29:50.519173 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:50.518764    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm" event={"ID":"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7","Type":"ContainerDied","Data":"fae55daa3b63d839f1ed39b4cdcae71d8ec149cd380b3f3a5f1defc9b0926d34"}
Nov 03 20:29:50.519173 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:50.518833    5659 scope.go:122] "RemoveContainer" containerID="a4df89a64c2e5938735144b9bbb1875e2547e7eac136bdbe68d7c82f327b5b3f"
Nov 03 20:29:50.519357 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:50.519340    5659 scope.go:122] "RemoveContainer" containerID="fae55daa3b63d839f1ed39b4cdcae71d8ec149cd380b3f3a5f1defc9b0926d34"
Nov 03 20:29:50.519859 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:50.519830    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-rk2jm_odh-ai-gateway-infra(bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm" podUID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7"
Nov 03 20:29:50.536780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:50.536751    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"]
Nov 03 20:29:50.636515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:50.626922    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:50.961725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:50.961698    5659 scope.go:122] "RemoveContainer" containerID="a1ddc01153408448ef96f9e185de7f758a2d12bfcc8ee7847016e975f9b1da33"
Nov 03 20:29:51.368508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:51.351351    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:51.531718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:51.531676    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf" event={"ID":"b9f68a13-f72d-4ecb-a2a1-3263747ab43d","Type":"ContainerStarted","Data":"b7eb3917b681a1021ee59162a9175d63e66224025bb805ca56c600464edc1892"}
Nov 03 20:29:51.745284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:51.745257    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"
Nov 03 20:29:51.822143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:51.822104    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-openshift-ca\") pod \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\" (UID: \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\") "
Nov 03 20:29:51.822347 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:51.822203    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-kube-api-access-znql2\" (UniqueName: \"kubernetes.io/projected/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-kube-api-access-znql2\") pod \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\" (UID: \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\") "
Nov 03 20:29:51.822405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:51.822364    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-xks-ca\") pod \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\" (UID: \"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7\") "
Nov 03 20:29:51.822635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:51.822602    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-openshift-ca" pod "bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" (UID: "bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:51.823290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:51.823266    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:51.825179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:51.825135    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-kube-api-access-znql2" pod "bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" (UID: "bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7"). InnerVolumeSpecName "kube-api-access-znql2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:51.825179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:51.825147    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-xks-ca" pod "bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" (UID: "bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:51.925420 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:51.925383    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znql2\" (UniqueName: \"kubernetes.io/projected/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-kube-api-access-znql2\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:51.925420 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:51.925415    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:52.035842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.035808    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:52.341450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.341364    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:52.481824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.481787    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:52.531297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.531264    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"]
Nov 03 20:29:52.532440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532412    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d1c367cc-759a-41c1-8640-00cf7b3b543d" containerName="revoke-keys"
Nov 03 20:29:52.532440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532441    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c367cc-759a-41c1-8640-00cf7b3b543d" containerName="revoke-keys"
Nov 03 20:29:52.533006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532454    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" containerName="revoke-keys"
Nov 03 20:29:52.533006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532462    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" containerName="revoke-keys"
Nov 03 20:29:52.533006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532498    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d1c367cc-759a-41c1-8640-00cf7b3b543d" containerName="revoke-keys"
Nov 03 20:29:52.533006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532507    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c367cc-759a-41c1-8640-00cf7b3b543d" containerName="revoke-keys"
Nov 03 20:29:52.533006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532520    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" containerName="revoke-keys"
Nov 03 20:29:52.533006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532529    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" containerName="revoke-keys"
Nov 03 20:29:52.533006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532813    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c367cc-759a-41c1-8640-00cf7b3b543d" containerName="revoke-keys"
Nov 03 20:29:52.533006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532827    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" containerName="revoke-keys"
Nov 03 20:29:52.533006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532840    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c367cc-759a-41c1-8640-00cf7b3b543d" containerName="revoke-keys"
Nov 03 20:29:52.533006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532848    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" containerName="revoke-keys"
Nov 03 20:29:52.533006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532862    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" containerName="revoke-keys"
Nov 03 20:29:52.533006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.532875    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c367cc-759a-41c1-8640-00cf7b3b543d" containerName="revoke-keys"
Nov 03 20:29:52.546556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.546524    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"
Nov 03 20:29:52.551569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.551532    5659 generic.go:356] "Generic (PLEG): container finished" podID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d" containerID="b7eb3917b681a1021ee59162a9175d63e66224025bb805ca56c600464edc1892" exitCode=22
Nov 03 20:29:52.557224 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.557184    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm" event={"ID":"bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7","Type":"ContainerDied","Data":"edfba1612b7f2409d8bb676b2c352ac24e352a5719e66ef0257daf424f63577a"}
Nov 03 20:29:52.557362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.557239    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"]
Nov 03 20:29:52.557362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.557283    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf" event={"ID":"b9f68a13-f72d-4ecb-a2a1-3263747ab43d","Type":"ContainerDied","Data":"b7eb3917b681a1021ee59162a9175d63e66224025bb805ca56c600464edc1892"}
Nov 03 20:29:52.557949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.557926    5659 scope.go:122] "RemoveContainer" containerID="b7eb3917b681a1021ee59162a9175d63e66224025bb805ca56c600464edc1892"
Nov 03 20:29:52.558666 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:52.558636    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-dhpvf_odh-ai-gateway-infra(b9f68a13-f72d-4ecb-a2a1-3263747ab43d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf" podUID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d"
Nov 03 20:29:52.559146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.559126    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"
Nov 03 20:29:52.560036 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.560017    5659 scope.go:122] "RemoveContainer" containerID="fae55daa3b63d839f1ed39b4cdcae71d8ec149cd380b3f3a5f1defc9b0926d34"
Nov 03 20:29:52.590001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.589968    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"]
Nov 03 20:29:52.595631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.595565    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"]
Nov 03 20:29:52.598163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.598138    5659 scope.go:122] "RemoveContainer" containerID="a1ddc01153408448ef96f9e185de7f758a2d12bfcc8ee7847016e975f9b1da33"
Nov 03 20:29:52.601498 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.601463    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-rk2jm"]
Nov 03 20:29:52.645265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.645223    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-j6m4t\" (UID: \"e4a8dc03-e370-4523-83ec-18288d6d1d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"
Nov 03 20:29:52.645434 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.645282    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkj2j\" (UniqueName: \"kubernetes.io/projected/e4a8dc03-e370-4523-83ec-18288d6d1d96-kube-api-access-nkj2j\") pod \"maas-api-revoke-sub-5bac6f1e-j6m4t\" (UID: \"e4a8dc03-e370-4523-83ec-18288d6d1d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"
Nov 03 20:29:52.645434 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.645312    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-j6m4t\" (UID: \"e4a8dc03-e370-4523-83ec-18288d6d1d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"
Nov 03 20:29:52.747311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.747266    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-j6m4t\" (UID: \"e4a8dc03-e370-4523-83ec-18288d6d1d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"
Nov 03 20:29:52.747540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.747317    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkj2j\" (UniqueName: \"kubernetes.io/projected/e4a8dc03-e370-4523-83ec-18288d6d1d96-kube-api-access-nkj2j\") pod \"maas-api-revoke-sub-5bac6f1e-j6m4t\" (UID: \"e4a8dc03-e370-4523-83ec-18288d6d1d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"
Nov 03 20:29:52.747540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.747347    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-j6m4t\" (UID: \"e4a8dc03-e370-4523-83ec-18288d6d1d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"
Nov 03 20:29:52.748085 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.748064    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-j6m4t\" (UID: \"e4a8dc03-e370-4523-83ec-18288d6d1d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"
Nov 03 20:29:52.749960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.749933    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-j6m4t\" (UID: \"e4a8dc03-e370-4523-83ec-18288d6d1d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"
Nov 03 20:29:52.759455 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.759429    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkj2j\" (UniqueName: \"kubernetes.io/projected/e4a8dc03-e370-4523-83ec-18288d6d1d96-kube-api-access-nkj2j\") pod \"maas-api-revoke-sub-5bac6f1e-j6m4t\" (UID: \"e4a8dc03-e370-4523-83ec-18288d6d1d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"
Nov 03 20:29:52.903268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.903168    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"
Nov 03 20:29:52.911529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:52.911063    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:53.290474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.290439    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"]
Nov 03 20:29:53.291374 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:29:53.291348    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a8dc03_e370_4523_83ec_18288d6d1d96.slice/crio-49e15fef987841156c0b949212fb9e0b50126290722ce460619930f410616d15 WatchSource:0}: Error finding container 49e15fef987841156c0b949212fb9e0b50126290722ce460619930f410616d15: Status 404 returned error can't find the container with id 49e15fef987841156c0b949212fb9e0b50126290722ce460619930f410616d15
Nov 03 20:29:53.391711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.391683    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:53.543943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.543856    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:53.564342 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.564301    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t" event={"ID":"e4a8dc03-e370-4523-83ec-18288d6d1d96","Type":"ContainerStarted","Data":"49e15fef987841156c0b949212fb9e0b50126290722ce460619930f410616d15"}
Nov 03 20:29:53.681946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.681908    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:53.796708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.796647    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"
Nov 03 20:29:53.843394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.843360    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:53.899505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.881178    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-xks-ca\") pod \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\" (UID: \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\") "
Nov 03 20:29:53.945704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.899865    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-openshift-ca\") pod \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\" (UID: \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\") "
Nov 03 20:29:53.957514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.956831    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-kube-api-access-djdjb\" (UniqueName: \"kubernetes.io/projected/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-kube-api-access-djdjb\") pod \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\" (UID: \"b9f68a13-f72d-4ecb-a2a1-3263747ab43d\") "
Nov 03 20:29:53.957780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.901004    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-openshift-ca" pod "b9f68a13-f72d-4ecb-a2a1-3263747ab43d" (UID: "b9f68a13-f72d-4ecb-a2a1-3263747ab43d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:53.957908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.901735    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-xks-ca" pod "b9f68a13-f72d-4ecb-a2a1-3263747ab43d" (UID: "b9f68a13-f72d-4ecb-a2a1-3263747ab43d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:53.965399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.965307    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-kube-api-access-djdjb" pod "b9f68a13-f72d-4ecb-a2a1-3263747ab43d" (UID: "b9f68a13-f72d-4ecb-a2a1-3263747ab43d"). InnerVolumeSpecName "kube-api-access-djdjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:53.979910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:53.979877    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" path="/var/lib/kubelet/pods/bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7/volumes"
Nov 03 20:29:54.060199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:54.060103    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:54.060199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:54.060140    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djdjb\" (UniqueName: \"kubernetes.io/projected/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-kube-api-access-djdjb\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:54.060199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:54.060157    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f68a13-f72d-4ecb-a2a1-3263747ab43d-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:54.581760 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:54.581729    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"
Nov 03 20:29:54.582261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:54.581763    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf" event={"ID":"b9f68a13-f72d-4ecb-a2a1-3263747ab43d","Type":"ContainerDied","Data":"5c2bbf68bb956c9dd5fedb68ea35b4d6f0937ac73b188754628fe816392dda06"}
Nov 03 20:29:54.582261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:54.581824    5659 scope.go:122] "RemoveContainer" containerID="b7eb3917b681a1021ee59162a9175d63e66224025bb805ca56c600464edc1892"
Nov 03 20:29:54.583988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:54.583950    5659 generic.go:356] "Generic (PLEG): container finished" podID="e4a8dc03-e370-4523-83ec-18288d6d1d96" containerID="36e58a56a46140e8b36b3c031c2c786eb24b322e886c96389478fb084e284d02" exitCode=22
Nov 03 20:29:54.583988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:54.583982    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t" event={"ID":"e4a8dc03-e370-4523-83ec-18288d6d1d96","Type":"ContainerDied","Data":"36e58a56a46140e8b36b3c031c2c786eb24b322e886c96389478fb084e284d02"}
Nov 03 20:29:54.584458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:54.584432    5659 scope.go:122] "RemoveContainer" containerID="36e58a56a46140e8b36b3c031c2c786eb24b322e886c96389478fb084e284d02"
Nov 03 20:29:54.608116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:54.607967    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"]
Nov 03 20:29:54.615215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:54.615188    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-dhpvf"]
Nov 03 20:29:54.637138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:54.637088    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:54.772448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:54.772417    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:55.078625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.078540    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"]
Nov 03 20:29:55.079852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.079829    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d1c367cc-759a-41c1-8640-00cf7b3b543d" containerName="revoke-keys"
Nov 03 20:29:55.079924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.079855    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c367cc-759a-41c1-8640-00cf7b3b543d" containerName="revoke-keys"
Nov 03 20:29:55.079924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.079869    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" containerName="revoke-keys"
Nov 03 20:29:55.079924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.079877    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9deb2b-ae86-44e4-b8b2-9cba6cdb77d7" containerName="revoke-keys"
Nov 03 20:29:55.079924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.079915    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d" containerName="revoke-keys"
Nov 03 20:29:55.079924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.079923    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d" containerName="revoke-keys"
Nov 03 20:29:55.080095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.079941    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d" containerName="revoke-keys"
Nov 03 20:29:55.080095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.079949    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d" containerName="revoke-keys"
Nov 03 20:29:55.080222 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.080210    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d" containerName="revoke-keys"
Nov 03 20:29:55.080301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.080226    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d" containerName="revoke-keys"
Nov 03 20:29:55.106537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.104467    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"]
Nov 03 20:29:55.106537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.104599    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"
Nov 03 20:29:55.185886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.185856    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-r76d5\" (UID: \"9e384d8e-346d-46fa-9381-f201482e7e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"
Nov 03 20:29:55.186060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.186045    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx8fk\" (UniqueName: \"kubernetes.io/projected/9e384d8e-346d-46fa-9381-f201482e7e8f-kube-api-access-mx8fk\") pod \"maas-api-revoke-sub-403cf98d-r76d5\" (UID: \"9e384d8e-346d-46fa-9381-f201482e7e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"
Nov 03 20:29:55.186251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.186240    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-r76d5\" (UID: \"9e384d8e-346d-46fa-9381-f201482e7e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"
Nov 03 20:29:55.288339 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.288295    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx8fk\" (UniqueName: \"kubernetes.io/projected/9e384d8e-346d-46fa-9381-f201482e7e8f-kube-api-access-mx8fk\") pod \"maas-api-revoke-sub-403cf98d-r76d5\" (UID: \"9e384d8e-346d-46fa-9381-f201482e7e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"
Nov 03 20:29:55.288639 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.288592    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-r76d5\" (UID: \"9e384d8e-346d-46fa-9381-f201482e7e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"
Nov 03 20:29:55.288700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.288643    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-r76d5\" (UID: \"9e384d8e-346d-46fa-9381-f201482e7e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"
Nov 03 20:29:55.289333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.289310    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-r76d5\" (UID: \"9e384d8e-346d-46fa-9381-f201482e7e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"
Nov 03 20:29:55.291307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.291288    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-r76d5\" (UID: \"9e384d8e-346d-46fa-9381-f201482e7e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"
Nov 03 20:29:55.300590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.300569    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx8fk\" (UniqueName: \"kubernetes.io/projected/9e384d8e-346d-46fa-9381-f201482e7e8f-kube-api-access-mx8fk\") pod \"maas-api-revoke-sub-403cf98d-r76d5\" (UID: \"9e384d8e-346d-46fa-9381-f201482e7e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"
Nov 03 20:29:55.330645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.330580    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:55.430796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.430762    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"
Nov 03 20:29:55.601691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.601605    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t" event={"ID":"e4a8dc03-e370-4523-83ec-18288d6d1d96","Type":"ContainerStarted","Data":"4dc5626400cd9b588b5f8d2c804882e661bc2faf53d2333fdf3a1fe6c9d3de64"}
Nov 03 20:29:55.622743 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.622688    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t" podStartSLOduration=3.622670666 podStartE2EDuration="3.622670666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:55.620338641 +0000 UTC m=+3147.775241062" watchObservedRunningTime="2026-11-03 20:29:55.622670666 +0000 UTC m=+3147.777573086"
Nov 03 20:29:55.737335 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:29:55.737299    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e384d8e_346d_46fa_9381_f201482e7e8f.slice/crio-8ac3624e19b0a7e60aafa92b190b872f68958258cec3940780477e3efe38856a WatchSource:0}: Error finding container 8ac3624e19b0a7e60aafa92b190b872f68958258cec3940780477e3efe38856a: Status 404 returned error can't find the container with id 8ac3624e19b0a7e60aafa92b190b872f68958258cec3940780477e3efe38856a
Nov 03 20:29:55.737637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.737610    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"]
Nov 03 20:29:55.973575 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:55.973540    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d" path="/var/lib/kubelet/pods/b9f68a13-f72d-4ecb-a2a1-3263747ab43d/volumes"
Nov 03 20:29:56.194411 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:56.194322    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:56.612838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:56.612804    5659 generic.go:356] "Generic (PLEG): container finished" podID="e4a8dc03-e370-4523-83ec-18288d6d1d96" containerID="4dc5626400cd9b588b5f8d2c804882e661bc2faf53d2333fdf3a1fe6c9d3de64" exitCode=22
Nov 03 20:29:56.613261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:56.612891    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t" event={"ID":"e4a8dc03-e370-4523-83ec-18288d6d1d96","Type":"ContainerDied","Data":"4dc5626400cd9b588b5f8d2c804882e661bc2faf53d2333fdf3a1fe6c9d3de64"}
Nov 03 20:29:56.613261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:56.613000    5659 scope.go:122] "RemoveContainer" containerID="36e58a56a46140e8b36b3c031c2c786eb24b322e886c96389478fb084e284d02"
Nov 03 20:29:56.613261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:56.613236    5659 scope.go:122] "RemoveContainer" containerID="4dc5626400cd9b588b5f8d2c804882e661bc2faf53d2333fdf3a1fe6c9d3de64"
Nov 03 20:29:56.613660 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:56.613640    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-j6m4t_odh-ai-gateway-infra(e4a8dc03-e370-4523-83ec-18288d6d1d96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t" podUID="e4a8dc03-e370-4523-83ec-18288d6d1d96"
Nov 03 20:29:56.615366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:56.615336    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5" event={"ID":"9e384d8e-346d-46fa-9381-f201482e7e8f","Type":"ContainerStarted","Data":"1d90bd62548d97a20817cdd52bb741a981cce5cad7207b8e2ea0a2adb94762f9"}
Nov 03 20:29:56.615531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:56.615377    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5" event={"ID":"9e384d8e-346d-46fa-9381-f201482e7e8f","Type":"ContainerStarted","Data":"8ac3624e19b0a7e60aafa92b190b872f68958258cec3940780477e3efe38856a"}
Nov 03 20:29:56.656638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:56.656573    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5" podStartSLOduration=1.656558547 podStartE2EDuration="1.656558547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:56.653329426 +0000 UTC m=+3148.808231845" watchObservedRunningTime="2026-11-03 20:29:56.656558547 +0000 UTC m=+3148.811460969"
Nov 03 20:29:56.839352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:56.839320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:57.498842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:57.498786    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:57.627176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:57.627135    5659 scope.go:122] "RemoveContainer" containerID="4dc5626400cd9b588b5f8d2c804882e661bc2faf53d2333fdf3a1fe6c9d3de64"
Nov 03 20:29:57.627745 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:57.627717    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-j6m4t_odh-ai-gateway-infra(e4a8dc03-e370-4523-83ec-18288d6d1d96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t" podUID="e4a8dc03-e370-4523-83ec-18288d6d1d96"
Nov 03 20:29:57.628547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:57.628523    5659 generic.go:356] "Generic (PLEG): container finished" podID="9e384d8e-346d-46fa-9381-f201482e7e8f" containerID="1d90bd62548d97a20817cdd52bb741a981cce5cad7207b8e2ea0a2adb94762f9" exitCode=22
Nov 03 20:29:57.628668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:57.628643    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5" event={"ID":"9e384d8e-346d-46fa-9381-f201482e7e8f","Type":"ContainerDied","Data":"1d90bd62548d97a20817cdd52bb741a981cce5cad7207b8e2ea0a2adb94762f9"}
Nov 03 20:29:57.628931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:57.628916    5659 scope.go:122] "RemoveContainer" containerID="1d90bd62548d97a20817cdd52bb741a981cce5cad7207b8e2ea0a2adb94762f9"
Nov 03 20:29:58.254616 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:58.254579    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:58.641105 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:58.641015    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5" event={"ID":"9e384d8e-346d-46fa-9381-f201482e7e8f","Type":"ContainerStarted","Data":"961882476b0599a83682f9387c5facec8ad25daea429ec2380461ebb9782cd30"}
Nov 03 20:29:58.934137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:58.934052    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:59.640421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:59.640382    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:59.737606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:59.731712    5659 generic.go:356] "Generic (PLEG): container finished" podID="9e384d8e-346d-46fa-9381-f201482e7e8f" containerID="961882476b0599a83682f9387c5facec8ad25daea429ec2380461ebb9782cd30" exitCode=22
Nov 03 20:29:59.737606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:59.732006    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5" event={"ID":"9e384d8e-346d-46fa-9381-f201482e7e8f","Type":"ContainerDied","Data":"961882476b0599a83682f9387c5facec8ad25daea429ec2380461ebb9782cd30"}
Nov 03 20:29:59.737606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:59.732112    5659 scope.go:122] "RemoveContainer" containerID="1d90bd62548d97a20817cdd52bb741a981cce5cad7207b8e2ea0a2adb94762f9"
Nov 03 20:29:59.737606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:29:59.733180    5659 scope.go:122] "RemoveContainer" containerID="961882476b0599a83682f9387c5facec8ad25daea429ec2380461ebb9782cd30"
Nov 03 20:29:59.737606 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:29:59.734008    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-r76d5_odh-ai-gateway-infra(9e384d8e-346d-46fa-9381-f201482e7e8f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5" podUID="9e384d8e-346d-46fa-9381-f201482e7e8f"
Nov 03 20:30:00.140415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.140383    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"]
Nov 03 20:30:00.141301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.141280    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d" containerName="revoke-keys"
Nov 03 20:30:00.141301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.141299    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d" containerName="revoke-keys"
Nov 03 20:30:00.141475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.141460    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f68a13-f72d-4ecb-a2a1-3263747ab43d" containerName="revoke-keys"
Nov 03 20:30:00.144981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.144940    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"
Nov 03 20:30:00.145721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.145695    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"]
Nov 03 20:30:00.149825 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.149804    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"
Nov 03 20:30:00.153188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.153169    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"]
Nov 03 20:30:00.159729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.159705    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"
Nov 03 20:30:00.160596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.160575    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"]
Nov 03 20:30:00.166402 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.166378    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"
Nov 03 20:30:00.168824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.167594    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"]
Nov 03 20:30:00.176795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.176223    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"]
Nov 03 20:30:00.192584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.192555    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"]
Nov 03 20:30:00.198842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.198813    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"]
Nov 03 20:30:00.298610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.298569    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b94007-86d5-47b9-8e7a-e439fd78fb39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4\" (UID: \"75b94007-86d5-47b9-8e7a-e439fd78fb39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"
Nov 03 20:30:00.298770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.298634    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzn4q\" (UniqueName: \"kubernetes.io/projected/8b99835e-da15-4f01-8df1-da9b74e76ae9-kube-api-access-jzn4q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg\" (UID: \"8b99835e-da15-4f01-8df1-da9b74e76ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"
Nov 03 20:30:00.298770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.298715    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b99835e-da15-4f01-8df1-da9b74e76ae9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg\" (UID: \"8b99835e-da15-4f01-8df1-da9b74e76ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"
Nov 03 20:30:00.298770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.298768    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ddxf\" (UniqueName: \"kubernetes.io/projected/a6a06ee0-a5ae-4986-a57d-ec34804041ed-kube-api-access-4ddxf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272\" (UID: \"a6a06ee0-a5ae-4986-a57d-ec34804041ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"
Nov 03 20:30:00.298943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.298846    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db088bce-259a-464c-953c-c2fe4d33582a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn\" (UID: \"db088bce-259a-464c-953c-c2fe4d33582a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"
Nov 03 20:30:00.298943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.298864    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4zdg\" (UniqueName: \"kubernetes.io/projected/75b94007-86d5-47b9-8e7a-e439fd78fb39-kube-api-access-v4zdg\") pod \"maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4\" (UID: \"75b94007-86d5-47b9-8e7a-e439fd78fb39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"
Nov 03 20:30:00.299150 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.299137    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a06ee0-a5ae-4986-a57d-ec34804041ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272\" (UID: \"a6a06ee0-a5ae-4986-a57d-ec34804041ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"
Nov 03 20:30:00.299194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.299177    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t79q\" (UniqueName: \"kubernetes.io/projected/db088bce-259a-464c-953c-c2fe4d33582a-kube-api-access-8t79q\") pod \"maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn\" (UID: \"db088bce-259a-464c-953c-c2fe4d33582a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"
Nov 03 20:30:00.400643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.400553    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a06ee0-a5ae-4986-a57d-ec34804041ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272\" (UID: \"a6a06ee0-a5ae-4986-a57d-ec34804041ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"
Nov 03 20:30:00.400824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.400716    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t79q\" (UniqueName: \"kubernetes.io/projected/db088bce-259a-464c-953c-c2fe4d33582a-kube-api-access-8t79q\") pod \"maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn\" (UID: \"db088bce-259a-464c-953c-c2fe4d33582a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"
Nov 03 20:30:00.400824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.400789    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b94007-86d5-47b9-8e7a-e439fd78fb39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4\" (UID: \"75b94007-86d5-47b9-8e7a-e439fd78fb39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"
Nov 03 20:30:00.400975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.400826    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzn4q\" (UniqueName: \"kubernetes.io/projected/8b99835e-da15-4f01-8df1-da9b74e76ae9-kube-api-access-jzn4q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg\" (UID: \"8b99835e-da15-4f01-8df1-da9b74e76ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"
Nov 03 20:30:00.400975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.400855    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b99835e-da15-4f01-8df1-da9b74e76ae9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg\" (UID: \"8b99835e-da15-4f01-8df1-da9b74e76ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"
Nov 03 20:30:00.400975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.400904    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ddxf\" (UniqueName: \"kubernetes.io/projected/a6a06ee0-a5ae-4986-a57d-ec34804041ed-kube-api-access-4ddxf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272\" (UID: \"a6a06ee0-a5ae-4986-a57d-ec34804041ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"
Nov 03 20:30:00.400975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.400937    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db088bce-259a-464c-953c-c2fe4d33582a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn\" (UID: \"db088bce-259a-464c-953c-c2fe4d33582a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"
Nov 03 20:30:00.400975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.400966    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4zdg\" (UniqueName: \"kubernetes.io/projected/75b94007-86d5-47b9-8e7a-e439fd78fb39-kube-api-access-v4zdg\") pod \"maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4\" (UID: \"75b94007-86d5-47b9-8e7a-e439fd78fb39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"
Nov 03 20:30:00.401574 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.401467    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a06ee0-a5ae-4986-a57d-ec34804041ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272\" (UID: \"a6a06ee0-a5ae-4986-a57d-ec34804041ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"
Nov 03 20:30:00.401715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.401658    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b94007-86d5-47b9-8e7a-e439fd78fb39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4\" (UID: \"75b94007-86d5-47b9-8e7a-e439fd78fb39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"
Nov 03 20:30:00.401805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.401782    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b99835e-da15-4f01-8df1-da9b74e76ae9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg\" (UID: \"8b99835e-da15-4f01-8df1-da9b74e76ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"
Nov 03 20:30:00.401869 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.401819    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db088bce-259a-464c-953c-c2fe4d33582a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn\" (UID: \"db088bce-259a-464c-953c-c2fe4d33582a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"
Nov 03 20:30:00.421003 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.420974    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ddxf\" (UniqueName: \"kubernetes.io/projected/a6a06ee0-a5ae-4986-a57d-ec34804041ed-kube-api-access-4ddxf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272\" (UID: \"a6a06ee0-a5ae-4986-a57d-ec34804041ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"
Nov 03 20:30:00.423835 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.423813    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzn4q\" (UniqueName: \"kubernetes.io/projected/8b99835e-da15-4f01-8df1-da9b74e76ae9-kube-api-access-jzn4q\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg\" (UID: \"8b99835e-da15-4f01-8df1-da9b74e76ae9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"
Nov 03 20:30:00.423973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.423873    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t79q\" (UniqueName: \"kubernetes.io/projected/db088bce-259a-464c-953c-c2fe4d33582a-kube-api-access-8t79q\") pod \"maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn\" (UID: \"db088bce-259a-464c-953c-c2fe4d33582a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"
Nov 03 20:30:00.424042 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.423996    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4zdg\" (UniqueName: \"kubernetes.io/projected/75b94007-86d5-47b9-8e7a-e439fd78fb39-kube-api-access-v4zdg\") pod \"maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4\" (UID: \"75b94007-86d5-47b9-8e7a-e439fd78fb39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"
Nov 03 20:30:00.464552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.464361    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:00.482115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.481796    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"
Nov 03 20:30:00.496889 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.496843    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"
Nov 03 20:30:00.535419 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.534881    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"
Nov 03 20:30:00.535419 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.535138    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"
Nov 03 20:30:00.749086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.749060    5659 scope.go:122] "RemoveContainer" containerID="961882476b0599a83682f9387c5facec8ad25daea429ec2380461ebb9782cd30"
Nov 03 20:30:00.749616 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:00.749574    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-r76d5_odh-ai-gateway-infra(9e384d8e-346d-46fa-9381-f201482e7e8f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5" podUID="9e384d8e-346d-46fa-9381-f201482e7e8f"
Nov 03 20:30:00.842169 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:00.842136    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b94007_86d5_47b9_8e7a_e439fd78fb39.slice/crio-baac2bc193b38611cfef13d8f2ef88580aa457da6e030d56f5b4cdb529a5cd59 WatchSource:0}: Error finding container baac2bc193b38611cfef13d8f2ef88580aa457da6e030d56f5b4cdb529a5cd59: Status 404 returned error can't find the container with id baac2bc193b38611cfef13d8f2ef88580aa457da6e030d56f5b4cdb529a5cd59
Nov 03 20:30:00.845810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.845711    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"]
Nov 03 20:30:00.862629 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:00.862591    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb088bce_259a_464c_953c_c2fe4d33582a.slice/crio-c2a7edd3ab82c5354682d7b6506bf39830d86fb73bdca6086f749a79e4268cba WatchSource:0}: Error finding container c2a7edd3ab82c5354682d7b6506bf39830d86fb73bdca6086f749a79e4268cba: Status 404 returned error can't find the container with id c2a7edd3ab82c5354682d7b6506bf39830d86fb73bdca6086f749a79e4268cba
Nov 03 20:30:00.863258 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.863226    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"]
Nov 03 20:30:00.909181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.909156    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"]
Nov 03 20:30:00.920797 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:00.920767    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a06ee0_a5ae_4986_a57d_ec34804041ed.slice/crio-9810ab93039ef006c832e7a490fe25b1d88ad9bb1fa2ebe64d23a3ba89435ad9 WatchSource:0}: Error finding container 9810ab93039ef006c832e7a490fe25b1d88ad9bb1fa2ebe64d23a3ba89435ad9: Status 404 returned error can't find the container with id 9810ab93039ef006c832e7a490fe25b1d88ad9bb1fa2ebe64d23a3ba89435ad9
Nov 03 20:30:00.929223 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:00.929193    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b99835e_da15_4f01_8df1_da9b74e76ae9.slice/crio-0cb52c55257b19752f7124fe8d62f7c633a7883026b11ace0c1e0fb543014a41 WatchSource:0}: Error finding container 0cb52c55257b19752f7124fe8d62f7c633a7883026b11ace0c1e0fb543014a41: Status 404 returned error can't find the container with id 0cb52c55257b19752f7124fe8d62f7c633a7883026b11ace0c1e0fb543014a41
Nov 03 20:30:00.931541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:00.931515    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"]
Nov 03 20:30:01.238672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.238640    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:01.763533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.763476    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272" event={"ID":"a6a06ee0-a5ae-4986-a57d-ec34804041ed","Type":"ContainerStarted","Data":"fa7da1ce5ec9566f544b3022a27da42f2f822bdfc142c0e9b607aed50ff7c5cc"}
Nov 03 20:30:01.764040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.763565    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272" event={"ID":"a6a06ee0-a5ae-4986-a57d-ec34804041ed","Type":"ContainerStarted","Data":"9810ab93039ef006c832e7a490fe25b1d88ad9bb1fa2ebe64d23a3ba89435ad9"}
Nov 03 20:30:01.765548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.765515    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4" event={"ID":"75b94007-86d5-47b9-8e7a-e439fd78fb39","Type":"ContainerStarted","Data":"8492a5d0f306f7a9b707374d6d27a778e0c4edb5cac327504f83e7f041e5cccd"}
Nov 03 20:30:01.765704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.765565    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4" event={"ID":"75b94007-86d5-47b9-8e7a-e439fd78fb39","Type":"ContainerStarted","Data":"baac2bc193b38611cfef13d8f2ef88580aa457da6e030d56f5b4cdb529a5cd59"}
Nov 03 20:30:01.767674 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.767640    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg" event={"ID":"8b99835e-da15-4f01-8df1-da9b74e76ae9","Type":"ContainerStarted","Data":"a80017db0218ce442e0ec551838ab3a047e2bac48c16e0159bfccad813dd6874"}
Nov 03 20:30:01.767854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.767697    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg" event={"ID":"8b99835e-da15-4f01-8df1-da9b74e76ae9","Type":"ContainerStarted","Data":"0cb52c55257b19752f7124fe8d62f7c633a7883026b11ace0c1e0fb543014a41"}
Nov 03 20:30:01.769475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.769450    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn" event={"ID":"db088bce-259a-464c-953c-c2fe4d33582a","Type":"ContainerStarted","Data":"e6a6e65eb40bc60dbfc721ae9f6b2fd2344c37ef0a64f1026b527efa35efe01f"}
Nov 03 20:30:01.769475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.769513    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn" event={"ID":"db088bce-259a-464c-953c-c2fe4d33582a","Type":"ContainerStarted","Data":"c2a7edd3ab82c5354682d7b6506bf39830d86fb73bdca6086f749a79e4268cba"}
Nov 03 20:30:01.781810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.781752    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272" podStartSLOduration=1.7817353740000001 podStartE2EDuration="1.781735374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:01.780025177 +0000 UTC m=+3153.934927608" watchObservedRunningTime="2026-11-03 20:30:01.781735374 +0000 UTC m=+3153.936637794"
Nov 03 20:30:01.807511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.803974    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg" podStartSLOduration=1.803956777 podStartE2EDuration="1.803956777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:01.796868099 +0000 UTC m=+3153.951770520" watchObservedRunningTime="2026-11-03 20:30:01.803956777 +0000 UTC m=+3153.958859196"
Nov 03 20:30:01.816900 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.816835    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn" podStartSLOduration=1.816817388 podStartE2EDuration="1.816817388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:01.815941765 +0000 UTC m=+3153.970844188" watchObservedRunningTime="2026-11-03 20:30:01.816817388 +0000 UTC m=+3153.971719806"
Nov 03 20:30:01.841289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.841220    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4" podStartSLOduration=1.8412060129999999 podStartE2EDuration="1.841206013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:01.838700686 +0000 UTC m=+3153.993603109" watchObservedRunningTime="2026-11-03 20:30:01.841206013 +0000 UTC m=+3153.996108431"
Nov 03 20:30:01.940687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:01.940648    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:02.647099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:02.647061    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:02.780471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:02.780432    5659 generic.go:356] "Generic (PLEG): container finished" podID="a6a06ee0-a5ae-4986-a57d-ec34804041ed" containerID="fa7da1ce5ec9566f544b3022a27da42f2f822bdfc142c0e9b607aed50ff7c5cc" exitCode=60
Nov 03 20:30:02.781015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:02.780513    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272" event={"ID":"a6a06ee0-a5ae-4986-a57d-ec34804041ed","Type":"ContainerDied","Data":"fa7da1ce5ec9566f544b3022a27da42f2f822bdfc142c0e9b607aed50ff7c5cc"}
Nov 03 20:30:02.781172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:02.781152    5659 scope.go:122] "RemoveContainer" containerID="fa7da1ce5ec9566f544b3022a27da42f2f822bdfc142c0e9b607aed50ff7c5cc"
Nov 03 20:30:02.782609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:02.782575    5659 generic.go:356] "Generic (PLEG): container finished" podID="75b94007-86d5-47b9-8e7a-e439fd78fb39" containerID="8492a5d0f306f7a9b707374d6d27a778e0c4edb5cac327504f83e7f041e5cccd" exitCode=60
Nov 03 20:30:02.782804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:02.782652    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4" event={"ID":"75b94007-86d5-47b9-8e7a-e439fd78fb39","Type":"ContainerDied","Data":"8492a5d0f306f7a9b707374d6d27a778e0c4edb5cac327504f83e7f041e5cccd"}
Nov 03 20:30:02.782959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:02.782942    5659 scope.go:122] "RemoveContainer" containerID="8492a5d0f306f7a9b707374d6d27a778e0c4edb5cac327504f83e7f041e5cccd"
Nov 03 20:30:02.785344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:02.785317    5659 generic.go:356] "Generic (PLEG): container finished" podID="8b99835e-da15-4f01-8df1-da9b74e76ae9" containerID="a80017db0218ce442e0ec551838ab3a047e2bac48c16e0159bfccad813dd6874" exitCode=60
Nov 03 20:30:02.785449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:02.785398    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg" event={"ID":"8b99835e-da15-4f01-8df1-da9b74e76ae9","Type":"ContainerDied","Data":"a80017db0218ce442e0ec551838ab3a047e2bac48c16e0159bfccad813dd6874"}
Nov 03 20:30:02.785758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:02.785739    5659 scope.go:122] "RemoveContainer" containerID="a80017db0218ce442e0ec551838ab3a047e2bac48c16e0159bfccad813dd6874"
Nov 03 20:30:02.787543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:02.787519    5659 generic.go:356] "Generic (PLEG): container finished" podID="db088bce-259a-464c-953c-c2fe4d33582a" containerID="e6a6e65eb40bc60dbfc721ae9f6b2fd2344c37ef0a64f1026b527efa35efe01f" exitCode=7
Nov 03 20:30:02.787658 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:02.787550    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn" event={"ID":"db088bce-259a-464c-953c-c2fe4d33582a","Type":"ContainerDied","Data":"e6a6e65eb40bc60dbfc721ae9f6b2fd2344c37ef0a64f1026b527efa35efe01f"}
Nov 03 20:30:02.787860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:02.787844    5659 scope.go:122] "RemoveContainer" containerID="e6a6e65eb40bc60dbfc721ae9f6b2fd2344c37ef0a64f1026b527efa35efe01f"
Nov 03 20:30:03.420812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:03.420729    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:03.798784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:03.798746    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg" event={"ID":"8b99835e-da15-4f01-8df1-da9b74e76ae9","Type":"ContainerStarted","Data":"5bb40208555e5a4ebf3d2d14590aaa03bf1a26d981c693a9c112e776a16ced2b"}
Nov 03 20:30:03.800916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:03.800885    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn" event={"ID":"db088bce-259a-464c-953c-c2fe4d33582a","Type":"ContainerStarted","Data":"a2e23e960b39efba7612d13852457148bfc93bf2ad1375c67d0a68d5c6a743bf"}
Nov 03 20:30:03.803108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:03.803081    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272" event={"ID":"a6a06ee0-a5ae-4986-a57d-ec34804041ed","Type":"ContainerStarted","Data":"e5c91ef4e495c5a0ac02222efcde04497e2735d06e734bff39a23ea40cfd0bf0"}
Nov 03 20:30:03.805061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:03.805040    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4" event={"ID":"75b94007-86d5-47b9-8e7a-e439fd78fb39","Type":"ContainerStarted","Data":"03c579c051957fe55ab7e20d8ca5cdb5d512cbd3522e19caf59ab0dd72e0543e"}
Nov 03 20:30:04.136084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.135925    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:04.818346 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.818306    5659 generic.go:356] "Generic (PLEG): container finished" podID="a6a06ee0-a5ae-4986-a57d-ec34804041ed" containerID="e5c91ef4e495c5a0ac02222efcde04497e2735d06e734bff39a23ea40cfd0bf0" exitCode=60
Nov 03 20:30:04.818852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.818383    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272" event={"ID":"a6a06ee0-a5ae-4986-a57d-ec34804041ed","Type":"ContainerDied","Data":"e5c91ef4e495c5a0ac02222efcde04497e2735d06e734bff39a23ea40cfd0bf0"}
Nov 03 20:30:04.818852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.818476    5659 scope.go:122] "RemoveContainer" containerID="fa7da1ce5ec9566f544b3022a27da42f2f822bdfc142c0e9b607aed50ff7c5cc"
Nov 03 20:30:04.818852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.818718    5659 scope.go:122] "RemoveContainer" containerID="e5c91ef4e495c5a0ac02222efcde04497e2735d06e734bff39a23ea40cfd0bf0"
Nov 03 20:30:04.819138 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:04.819107    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272_odh-ai-gateway-infra(a6a06ee0-a5ae-4986-a57d-ec34804041ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272" podUID="a6a06ee0-a5ae-4986-a57d-ec34804041ed"
Nov 03 20:30:04.820784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.820760    5659 generic.go:356] "Generic (PLEG): container finished" podID="75b94007-86d5-47b9-8e7a-e439fd78fb39" containerID="03c579c051957fe55ab7e20d8ca5cdb5d512cbd3522e19caf59ab0dd72e0543e" exitCode=60
Nov 03 20:30:04.820912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.820841    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4" event={"ID":"75b94007-86d5-47b9-8e7a-e439fd78fb39","Type":"ContainerDied","Data":"03c579c051957fe55ab7e20d8ca5cdb5d512cbd3522e19caf59ab0dd72e0543e"}
Nov 03 20:30:04.821100 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.821078    5659 scope.go:122] "RemoveContainer" containerID="03c579c051957fe55ab7e20d8ca5cdb5d512cbd3522e19caf59ab0dd72e0543e"
Nov 03 20:30:04.821451 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:04.821421    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4_odh-ai-gateway-infra(75b94007-86d5-47b9-8e7a-e439fd78fb39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4" podUID="75b94007-86d5-47b9-8e7a-e439fd78fb39"
Nov 03 20:30:04.823162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.823132    5659 generic.go:356] "Generic (PLEG): container finished" podID="8b99835e-da15-4f01-8df1-da9b74e76ae9" containerID="5bb40208555e5a4ebf3d2d14590aaa03bf1a26d981c693a9c112e776a16ced2b" exitCode=60
Nov 03 20:30:04.823294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.823192    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg" event={"ID":"8b99835e-da15-4f01-8df1-da9b74e76ae9","Type":"ContainerDied","Data":"5bb40208555e5a4ebf3d2d14590aaa03bf1a26d981c693a9c112e776a16ced2b"}
Nov 03 20:30:04.823567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.823550    5659 scope.go:122] "RemoveContainer" containerID="5bb40208555e5a4ebf3d2d14590aaa03bf1a26d981c693a9c112e776a16ced2b"
Nov 03 20:30:04.823888 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:04.823864    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg_odh-ai-gateway-infra(8b99835e-da15-4f01-8df1-da9b74e76ae9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg" podUID="8b99835e-da15-4f01-8df1-da9b74e76ae9"
Nov 03 20:30:04.825557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.825533    5659 generic.go:356] "Generic (PLEG): container finished" podID="db088bce-259a-464c-953c-c2fe4d33582a" containerID="a2e23e960b39efba7612d13852457148bfc93bf2ad1375c67d0a68d5c6a743bf" exitCode=7
Nov 03 20:30:04.825906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.825614    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn" event={"ID":"db088bce-259a-464c-953c-c2fe4d33582a","Type":"ContainerDied","Data":"a2e23e960b39efba7612d13852457148bfc93bf2ad1375c67d0a68d5c6a743bf"}
Nov 03 20:30:04.825906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.825830    5659 scope.go:122] "RemoveContainer" containerID="a2e23e960b39efba7612d13852457148bfc93bf2ad1375c67d0a68d5c6a743bf"
Nov 03 20:30:04.826171 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:04.826147    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn_odh-ai-gateway-infra(db088bce-259a-464c-953c-c2fe4d33582a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn" podUID="db088bce-259a-464c-953c-c2fe4d33582a"
Nov 03 20:30:04.885630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.881261    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:04.935009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.934977    5659 scope.go:122] "RemoveContainer" containerID="8492a5d0f306f7a9b707374d6d27a778e0c4edb5cac327504f83e7f041e5cccd"
Nov 03 20:30:04.981237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:04.981210    5659 scope.go:122] "RemoveContainer" containerID="a80017db0218ce442e0ec551838ab3a047e2bac48c16e0159bfccad813dd6874"
Nov 03 20:30:05.025224 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:05.025195    5659 scope.go:122] "RemoveContainer" containerID="e6a6e65eb40bc60dbfc721ae9f6b2fd2344c37ef0a64f1026b527efa35efe01f"
Nov 03 20:30:05.752290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:05.752249    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:05.885184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:05.885153    5659 scope.go:122] "RemoveContainer" containerID="e5c91ef4e495c5a0ac02222efcde04497e2735d06e734bff39a23ea40cfd0bf0"
Nov 03 20:30:05.885766 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:05.885740    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272_odh-ai-gateway-infra(a6a06ee0-a5ae-4986-a57d-ec34804041ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272" podUID="a6a06ee0-a5ae-4986-a57d-ec34804041ed"
Nov 03 20:30:05.887420 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:05.887393    5659 scope.go:122] "RemoveContainer" containerID="03c579c051957fe55ab7e20d8ca5cdb5d512cbd3522e19caf59ab0dd72e0543e"
Nov 03 20:30:05.887798 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:05.887773    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4_odh-ai-gateway-infra(75b94007-86d5-47b9-8e7a-e439fd78fb39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4" podUID="75b94007-86d5-47b9-8e7a-e439fd78fb39"
Nov 03 20:30:05.889799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:05.889783    5659 scope.go:122] "RemoveContainer" containerID="5bb40208555e5a4ebf3d2d14590aaa03bf1a26d981c693a9c112e776a16ced2b"
Nov 03 20:30:05.890080 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:05.890059    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg_odh-ai-gateway-infra(8b99835e-da15-4f01-8df1-da9b74e76ae9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg" podUID="8b99835e-da15-4f01-8df1-da9b74e76ae9"
Nov 03 20:30:05.891607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:05.891587    5659 scope.go:122] "RemoveContainer" containerID="a2e23e960b39efba7612d13852457148bfc93bf2ad1375c67d0a68d5c6a743bf"
Nov 03 20:30:05.891888 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:05.891863    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn_odh-ai-gateway-infra(db088bce-259a-464c-953c-c2fe4d33582a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn" podUID="db088bce-259a-464c-953c-c2fe4d33582a"
Nov 03 20:30:06.543648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:06.543599    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:06.961448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:06.961417    5659 scope.go:122] "RemoveContainer" containerID="4dc5626400cd9b588b5f8d2c804882e661bc2faf53d2333fdf3a1fe6c9d3de64"
Nov 03 20:30:07.224040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:07.223895    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:07.913400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:07.913352    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t" event={"ID":"e4a8dc03-e370-4523-83ec-18288d6d1d96","Type":"ContainerStarted","Data":"73142de31d6919bade973c6646436b032c797a42b8677c0b6eabc0a3b97b0af7"}
Nov 03 20:30:07.954133 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:07.954085    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:08.624131 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:08.624093    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:08.927183 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:08.927049    5659 generic.go:356] "Generic (PLEG): container finished" podID="e4a8dc03-e370-4523-83ec-18288d6d1d96" containerID="73142de31d6919bade973c6646436b032c797a42b8677c0b6eabc0a3b97b0af7" exitCode=22
Nov 03 20:30:08.927183 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:08.927125    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t" event={"ID":"e4a8dc03-e370-4523-83ec-18288d6d1d96","Type":"ContainerDied","Data":"73142de31d6919bade973c6646436b032c797a42b8677c0b6eabc0a3b97b0af7"}
Nov 03 20:30:08.927437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:08.927187    5659 scope.go:122] "RemoveContainer" containerID="4dc5626400cd9b588b5f8d2c804882e661bc2faf53d2333fdf3a1fe6c9d3de64"
Nov 03 20:30:08.927677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:08.927636    5659 scope.go:122] "RemoveContainer" containerID="73142de31d6919bade973c6646436b032c797a42b8677c0b6eabc0a3b97b0af7"
Nov 03 20:30:08.928084 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:08.928060    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-j6m4t_odh-ai-gateway-infra(e4a8dc03-e370-4523-83ec-18288d6d1d96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t" podUID="e4a8dc03-e370-4523-83ec-18288d6d1d96"
Nov 03 20:30:08.949181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:08.949145    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"]
Nov 03 20:30:08.961406 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:08.961377    5659 scope.go:122] "RemoveContainer" containerID="961882476b0599a83682f9387c5facec8ad25daea429ec2380461ebb9782cd30"
Nov 03 20:30:09.335920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:09.335884    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:09.942151 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:09.942120    5659 generic.go:356] "Generic (PLEG): container finished" podID="9e384d8e-346d-46fa-9381-f201482e7e8f" containerID="5842e55becb013d92d39375af06e135be7552f72927533184bbeeaa1e48bbbb7" exitCode=22
Nov 03 20:30:09.942606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:09.942198    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5" event={"ID":"9e384d8e-346d-46fa-9381-f201482e7e8f","Type":"ContainerDied","Data":"5842e55becb013d92d39375af06e135be7552f72927533184bbeeaa1e48bbbb7"}
Nov 03 20:30:09.942606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:09.942287    5659 scope.go:122] "RemoveContainer" containerID="961882476b0599a83682f9387c5facec8ad25daea429ec2380461ebb9782cd30"
Nov 03 20:30:09.942874 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:09.942846    5659 scope.go:122] "RemoveContainer" containerID="5842e55becb013d92d39375af06e135be7552f72927533184bbeeaa1e48bbbb7"
Nov 03 20:30:09.943577 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:09.943364    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-r76d5_odh-ai-gateway-infra(9e384d8e-346d-46fa-9381-f201482e7e8f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5" podUID="9e384d8e-346d-46fa-9381-f201482e7e8f"
Nov 03 20:30:10.104643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.099668    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:10.259095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.259054    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"
Nov 03 20:30:10.292617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.292469    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"]
Nov 03 20:30:10.293531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.293505    5659 cpu_manager.go:414] "containerMap: removing container" podUID="e4a8dc03-e370-4523-83ec-18288d6d1d96" containerName="revoke-keys"
Nov 03 20:30:10.293531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.293530    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a8dc03-e370-4523-83ec-18288d6d1d96" containerName="revoke-keys"
Nov 03 20:30:10.293725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.293552    5659 cpu_manager.go:414] "containerMap: removing container" podUID="e4a8dc03-e370-4523-83ec-18288d6d1d96" containerName="revoke-keys"
Nov 03 20:30:10.293725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.293560    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a8dc03-e370-4523-83ec-18288d6d1d96" containerName="revoke-keys"
Nov 03 20:30:10.293854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.293842    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a8dc03-e370-4523-83ec-18288d6d1d96" containerName="revoke-keys"
Nov 03 20:30:10.293914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.293859    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a8dc03-e370-4523-83ec-18288d6d1d96" containerName="revoke-keys"
Nov 03 20:30:10.311394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.311363    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"
Nov 03 20:30:10.321139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.321106    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"]
Nov 03 20:30:10.361636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.361540    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-openshift-ca\") pod \"e4a8dc03-e370-4523-83ec-18288d6d1d96\" (UID: \"e4a8dc03-e370-4523-83ec-18288d6d1d96\") "
Nov 03 20:30:10.361636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.361590    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a8dc03-e370-4523-83ec-18288d6d1d96-kube-api-access-nkj2j\" (UniqueName: \"kubernetes.io/projected/e4a8dc03-e370-4523-83ec-18288d6d1d96-kube-api-access-nkj2j\") pod \"e4a8dc03-e370-4523-83ec-18288d6d1d96\" (UID: \"e4a8dc03-e370-4523-83ec-18288d6d1d96\") "
Nov 03 20:30:10.361881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.361644    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-xks-ca\") pod \"e4a8dc03-e370-4523-83ec-18288d6d1d96\" (UID: \"e4a8dc03-e370-4523-83ec-18288d6d1d96\") "
Nov 03 20:30:10.362100 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.362072    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d88c4-b3c0-49d1-b799-7c746abdef13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp\" (UID: \"659d88c4-b3c0-49d1-b799-7c746abdef13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"
Nov 03 20:30:10.362173 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.362151    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-openshift-ca" pod "e4a8dc03-e370-4523-83ec-18288d6d1d96" (UID: "e4a8dc03-e370-4523-83ec-18288d6d1d96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:10.362827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.362802    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn5pk\" (UniqueName: \"kubernetes.io/projected/659d88c4-b3c0-49d1-b799-7c746abdef13-kube-api-access-xn5pk\") pod \"maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp\" (UID: \"659d88c4-b3c0-49d1-b799-7c746abdef13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"
Nov 03 20:30:10.363127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.363107    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:10.367573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.365061    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a8dc03-e370-4523-83ec-18288d6d1d96-kube-api-access-nkj2j" pod "e4a8dc03-e370-4523-83ec-18288d6d1d96" (UID: "e4a8dc03-e370-4523-83ec-18288d6d1d96"). InnerVolumeSpecName "kube-api-access-nkj2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:10.367573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.365644    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-xks-ca" pod "e4a8dc03-e370-4523-83ec-18288d6d1d96" (UID: "e4a8dc03-e370-4523-83ec-18288d6d1d96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:10.465185 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.465145    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn5pk\" (UniqueName: \"kubernetes.io/projected/659d88c4-b3c0-49d1-b799-7c746abdef13-kube-api-access-xn5pk\") pod \"maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp\" (UID: \"659d88c4-b3c0-49d1-b799-7c746abdef13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"
Nov 03 20:30:10.465412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.465291    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d88c4-b3c0-49d1-b799-7c746abdef13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp\" (UID: \"659d88c4-b3c0-49d1-b799-7c746abdef13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"
Nov 03 20:30:10.465793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.465763    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkj2j\" (UniqueName: \"kubernetes.io/projected/e4a8dc03-e370-4523-83ec-18288d6d1d96-kube-api-access-nkj2j\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:10.465871 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.465817    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a8dc03-e370-4523-83ec-18288d6d1d96-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:10.466058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.466040    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d88c4-b3c0-49d1-b799-7c746abdef13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp\" (UID: \"659d88c4-b3c0-49d1-b799-7c746abdef13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"
Nov 03 20:30:10.477737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.477705    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn5pk\" (UniqueName: \"kubernetes.io/projected/659d88c4-b3c0-49d1-b799-7c746abdef13-kube-api-access-xn5pk\") pod \"maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp\" (UID: \"659d88c4-b3c0-49d1-b799-7c746abdef13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"
Nov 03 20:30:10.644885 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.644795    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"
Nov 03 20:30:10.764300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.764266    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:10.956099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.956070    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"]
Nov 03 20:30:10.957542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.957513    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"
Nov 03 20:30:10.957674 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.957540    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t" event={"ID":"e4a8dc03-e370-4523-83ec-18288d6d1d96","Type":"ContainerDied","Data":"49e15fef987841156c0b949212fb9e0b50126290722ce460619930f410616d15"}
Nov 03 20:30:10.957674 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.957599    5659 scope.go:122] "RemoveContainer" containerID="73142de31d6919bade973c6646436b032c797a42b8677c0b6eabc0a3b97b0af7"
Nov 03 20:30:10.968992 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:10.968191    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659d88c4_b3c0_49d1_b799_7c746abdef13.slice/crio-595e970565555b9f41209f01da3b0835b27add489f0bcbf31c79d0e22e8ff621 WatchSource:0}: Error finding container 595e970565555b9f41209f01da3b0835b27add489f0bcbf31c79d0e22e8ff621: Status 404 returned error can't find the container with id 595e970565555b9f41209f01da3b0835b27add489f0bcbf31c79d0e22e8ff621
Nov 03 20:30:10.987867 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.987841    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"]
Nov 03 20:30:10.992812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:10.992786    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-j6m4t"]
Nov 03 20:30:11.000421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.000394    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"]
Nov 03 20:30:11.232947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.232868    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"
Nov 03 20:30:11.277999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.277962    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"]
Nov 03 20:30:11.279359 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.279328    5659 cpu_manager.go:414] "containerMap: removing container" podUID="9e384d8e-346d-46fa-9381-f201482e7e8f" containerName="revoke-keys"
Nov 03 20:30:11.279359 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.279357    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e384d8e-346d-46fa-9381-f201482e7e8f" containerName="revoke-keys"
Nov 03 20:30:11.279610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.279373    5659 cpu_manager.go:414] "containerMap: removing container" podUID="e4a8dc03-e370-4523-83ec-18288d6d1d96" containerName="revoke-keys"
Nov 03 20:30:11.279610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.279382    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a8dc03-e370-4523-83ec-18288d6d1d96" containerName="revoke-keys"
Nov 03 20:30:11.279610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.279512    5659 cpu_manager.go:414] "containerMap: removing container" podUID="9e384d8e-346d-46fa-9381-f201482e7e8f" containerName="revoke-keys"
Nov 03 20:30:11.279610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.279521    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e384d8e-346d-46fa-9381-f201482e7e8f" containerName="revoke-keys"
Nov 03 20:30:11.279905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.279726    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e384d8e-346d-46fa-9381-f201482e7e8f" containerName="revoke-keys"
Nov 03 20:30:11.279905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.279756    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e384d8e-346d-46fa-9381-f201482e7e8f" containerName="revoke-keys"
Nov 03 20:30:11.279905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.279769    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e384d8e-346d-46fa-9381-f201482e7e8f" containerName="revoke-keys"
Nov 03 20:30:11.279905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.279782    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a8dc03-e370-4523-83ec-18288d6d1d96" containerName="revoke-keys"
Nov 03 20:30:11.286912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.286843    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-openshift-ca\") pod \"9e384d8e-346d-46fa-9381-f201482e7e8f\" (UID: \"9e384d8e-346d-46fa-9381-f201482e7e8f\") "
Nov 03 20:30:11.287086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.286913    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e384d8e-346d-46fa-9381-f201482e7e8f-kube-api-access-mx8fk\" (UniqueName: \"kubernetes.io/projected/9e384d8e-346d-46fa-9381-f201482e7e8f-kube-api-access-mx8fk\") pod \"9e384d8e-346d-46fa-9381-f201482e7e8f\" (UID: \"9e384d8e-346d-46fa-9381-f201482e7e8f\") "
Nov 03 20:30:11.287158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.287104    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-xks-ca\") pod \"9e384d8e-346d-46fa-9381-f201482e7e8f\" (UID: \"9e384d8e-346d-46fa-9381-f201482e7e8f\") "
Nov 03 20:30:11.288024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.287977    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-openshift-ca" pod "9e384d8e-346d-46fa-9381-f201482e7e8f" (UID: "9e384d8e-346d-46fa-9381-f201482e7e8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:11.288181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.288156    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:11.290720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.290694    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-xks-ca" pod "9e384d8e-346d-46fa-9381-f201482e7e8f" (UID: "9e384d8e-346d-46fa-9381-f201482e7e8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:11.290835 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.290743    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e384d8e-346d-46fa-9381-f201482e7e8f-kube-api-access-mx8fk" pod "9e384d8e-346d-46fa-9381-f201482e7e8f" (UID: "9e384d8e-346d-46fa-9381-f201482e7e8f"). InnerVolumeSpecName "kube-api-access-mx8fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:11.307880 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.307834    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"]
Nov 03 20:30:11.308011 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.307936    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"
Nov 03 20:30:11.390144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.390106    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844f0645-fbd2-4496-9621-343de2fec107-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9\" (UID: \"844f0645-fbd2-4496-9621-343de2fec107\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"
Nov 03 20:30:11.390422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.390404    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwghk\" (UniqueName: \"kubernetes.io/projected/844f0645-fbd2-4496-9621-343de2fec107-kube-api-access-hwghk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9\" (UID: \"844f0645-fbd2-4496-9621-343de2fec107\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"
Nov 03 20:30:11.390625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.390605    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx8fk\" (UniqueName: \"kubernetes.io/projected/9e384d8e-346d-46fa-9381-f201482e7e8f-kube-api-access-mx8fk\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:11.390709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.390632    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e384d8e-346d-46fa-9381-f201482e7e8f-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:11.492228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.492157    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844f0645-fbd2-4496-9621-343de2fec107-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9\" (UID: \"844f0645-fbd2-4496-9621-343de2fec107\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"
Nov 03 20:30:11.492378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.492350    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwghk\" (UniqueName: \"kubernetes.io/projected/844f0645-fbd2-4496-9621-343de2fec107-kube-api-access-hwghk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9\" (UID: \"844f0645-fbd2-4496-9621-343de2fec107\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"
Nov 03 20:30:11.492795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.492773    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844f0645-fbd2-4496-9621-343de2fec107-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9\" (UID: \"844f0645-fbd2-4496-9621-343de2fec107\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"
Nov 03 20:30:11.504724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.504699    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwghk\" (UniqueName: \"kubernetes.io/projected/844f0645-fbd2-4496-9621-343de2fec107-kube-api-access-hwghk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9\" (UID: \"844f0645-fbd2-4496-9621-343de2fec107\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"
Nov 03 20:30:11.539236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.539193    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:11.664111 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.664075    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"
Nov 03 20:30:11.979410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.978935    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4a8dc03-e370-4523-83ec-18288d6d1d96" path="/var/lib/kubelet/pods/e4a8dc03-e370-4523-83ec-18288d6d1d96/volumes"
Nov 03 20:30:11.979833 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.979739    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"]
Nov 03 20:30:11.982878 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.982849    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"
Nov 03 20:30:11.983021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.982896    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5" event={"ID":"9e384d8e-346d-46fa-9381-f201482e7e8f","Type":"ContainerDied","Data":"8ac3624e19b0a7e60aafa92b190b872f68958258cec3940780477e3efe38856a"}
Nov 03 20:30:11.983021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.982947    5659 scope.go:122] "RemoveContainer" containerID="5842e55becb013d92d39375af06e135be7552f72927533184bbeeaa1e48bbbb7"
Nov 03 20:30:11.985636 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:11.985608    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844f0645_fbd2_4496_9621_343de2fec107.slice/crio-9a50b1c3ee4ba045283c56f3d60581f3652c0b999a835d0a1db0228077a79bc9 WatchSource:0}: Error finding container 9a50b1c3ee4ba045283c56f3d60581f3652c0b999a835d0a1db0228077a79bc9: Status 404 returned error can't find the container with id 9a50b1c3ee4ba045283c56f3d60581f3652c0b999a835d0a1db0228077a79bc9
Nov 03 20:30:11.987045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.986623    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp" event={"ID":"659d88c4-b3c0-49d1-b799-7c746abdef13","Type":"ContainerStarted","Data":"e4451e62eb786267b9f07c77af5a3780f32cfed815c5b1c7250ec8d5b3a89c57"}
Nov 03 20:30:11.987045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:11.986682    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp" event={"ID":"659d88c4-b3c0-49d1-b799-7c746abdef13","Type":"ContainerStarted","Data":"595e970565555b9f41209f01da3b0835b27add489f0bcbf31c79d0e22e8ff621"}
Nov 03 20:30:12.018694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:12.018649    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp" podStartSLOduration=12.018634533 podStartE2EDuration="12.018634533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:12.015445859 +0000 UTC m=+3164.170348276" watchObservedRunningTime="2026-11-03 20:30:12.018634533 +0000 UTC m=+3164.173536951"
Nov 03 20:30:12.038695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:12.038656    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"]
Nov 03 20:30:12.044109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:12.044082    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r76d5"]
Nov 03 20:30:12.328287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:12.328231    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:12.997653 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:12.997614    5659 generic.go:356] "Generic (PLEG): container finished" podID="659d88c4-b3c0-49d1-b799-7c746abdef13" containerID="e4451e62eb786267b9f07c77af5a3780f32cfed815c5b1c7250ec8d5b3a89c57" exitCode=60
Nov 03 20:30:12.998185 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:12.997712    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp" event={"ID":"659d88c4-b3c0-49d1-b799-7c746abdef13","Type":"ContainerDied","Data":"e4451e62eb786267b9f07c77af5a3780f32cfed815c5b1c7250ec8d5b3a89c57"}
Nov 03 20:30:12.998185 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:12.998129    5659 scope.go:122] "RemoveContainer" containerID="e4451e62eb786267b9f07c77af5a3780f32cfed815c5b1c7250ec8d5b3a89c57"
Nov 03 20:30:13.000050 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:13.000019    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9" event={"ID":"844f0645-fbd2-4496-9621-343de2fec107","Type":"ContainerStarted","Data":"858eb317a48e16758deba8c6f355c5d690f65cdd469e62bb2383265ea1b01717"}
Nov 03 20:30:13.000190 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:13.000087    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9" event={"ID":"844f0645-fbd2-4496-9621-343de2fec107","Type":"ContainerStarted","Data":"9a50b1c3ee4ba045283c56f3d60581f3652c0b999a835d0a1db0228077a79bc9"}
Nov 03 20:30:13.045740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:13.044170    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9" podStartSLOduration=13.044150463 podStartE2EDuration="13.044150463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:13.038803221 +0000 UTC m=+3165.193705643" watchObservedRunningTime="2026-11-03 20:30:13.044150463 +0000 UTC m=+3165.199052883"
Nov 03 20:30:13.137020 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:13.136987    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:13.944384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:13.938579    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:14.025624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:14.025589    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e384d8e-346d-46fa-9381-f201482e7e8f" path="/var/lib/kubelet/pods/9e384d8e-346d-46fa-9381-f201482e7e8f/volumes"
Nov 03 20:30:14.030404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:14.030375    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp" event={"ID":"659d88c4-b3c0-49d1-b799-7c746abdef13","Type":"ContainerStarted","Data":"8602769d5612cc2c0f2a528ee4b64f30e3a87b9148fe1837fd414b8244353d53"}
Nov 03 20:30:14.032375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:14.032344    5659 generic.go:356] "Generic (PLEG): container finished" podID="844f0645-fbd2-4496-9621-343de2fec107" containerID="858eb317a48e16758deba8c6f355c5d690f65cdd469e62bb2383265ea1b01717" exitCode=60
Nov 03 20:30:14.032668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:14.032408    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9" event={"ID":"844f0645-fbd2-4496-9621-343de2fec107","Type":"ContainerDied","Data":"858eb317a48e16758deba8c6f355c5d690f65cdd469e62bb2383265ea1b01717"}
Nov 03 20:30:14.032876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:14.032857    5659 scope.go:122] "RemoveContainer" containerID="858eb317a48e16758deba8c6f355c5d690f65cdd469e62bb2383265ea1b01717"
Nov 03 20:30:14.635203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:14.635107    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:14.962123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:14.962090    5659 scope.go:122] "RemoveContainer" containerID="5bb40208555e5a4ebf3d2d14590aaa03bf1a26d981c693a9c112e776a16ced2b"
Nov 03 20:30:14.962323 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:14.962300    5659 scope.go:122] "RemoveContainer" containerID="e5c91ef4e495c5a0ac02222efcde04497e2735d06e734bff39a23ea40cfd0bf0"
Nov 03 20:30:14.962467 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:14.962447    5659 scope.go:122] "RemoveContainer" containerID="03c579c051957fe55ab7e20d8ca5cdb5d512cbd3522e19caf59ab0dd72e0543e"
Nov 03 20:30:15.045859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:15.045825    5659 generic.go:356] "Generic (PLEG): container finished" podID="659d88c4-b3c0-49d1-b799-7c746abdef13" containerID="8602769d5612cc2c0f2a528ee4b64f30e3a87b9148fe1837fd414b8244353d53" exitCode=60
Nov 03 20:30:15.046313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:15.045913    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp" event={"ID":"659d88c4-b3c0-49d1-b799-7c746abdef13","Type":"ContainerDied","Data":"8602769d5612cc2c0f2a528ee4b64f30e3a87b9148fe1837fd414b8244353d53"}
Nov 03 20:30:15.046313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:15.045991    5659 scope.go:122] "RemoveContainer" containerID="e4451e62eb786267b9f07c77af5a3780f32cfed815c5b1c7250ec8d5b3a89c57"
Nov 03 20:30:15.046473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:15.046423    5659 scope.go:122] "RemoveContainer" containerID="8602769d5612cc2c0f2a528ee4b64f30e3a87b9148fe1837fd414b8244353d53"
Nov 03 20:30:15.046882 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:15.046856    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp_odh-ai-gateway-infra(659d88c4-b3c0-49d1-b799-7c746abdef13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp" podUID="659d88c4-b3c0-49d1-b799-7c746abdef13"
Nov 03 20:30:15.048621 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:15.048594    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9" event={"ID":"844f0645-fbd2-4496-9621-343de2fec107","Type":"ContainerStarted","Data":"8fe02ef24f61bd94f9fa933bcdb837e914c88f67e27d5d35e5a39ed7f4d1f662"}
Nov 03 20:30:15.262699 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:15.262663    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:30:15.415612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:15.415545    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:15.961722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:15.961686    5659 scope.go:122] "RemoveContainer" containerID="a2e23e960b39efba7612d13852457148bfc93bf2ad1375c67d0a68d5c6a743bf"
Nov 03 20:30:16.060244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:16.060214    5659 scope.go:122] "RemoveContainer" containerID="8602769d5612cc2c0f2a528ee4b64f30e3a87b9148fe1837fd414b8244353d53"
Nov 03 20:30:16.060717 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:16.060687    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp_odh-ai-gateway-infra(659d88c4-b3c0-49d1-b799-7c746abdef13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp" podUID="659d88c4-b3c0-49d1-b799-7c746abdef13"
Nov 03 20:30:16.062252 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:16.062223    5659 generic.go:356] "Generic (PLEG): container finished" podID="844f0645-fbd2-4496-9621-343de2fec107" containerID="8fe02ef24f61bd94f9fa933bcdb837e914c88f67e27d5d35e5a39ed7f4d1f662" exitCode=60
Nov 03 20:30:16.062395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:16.062305    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9" event={"ID":"844f0645-fbd2-4496-9621-343de2fec107","Type":"ContainerDied","Data":"8fe02ef24f61bd94f9fa933bcdb837e914c88f67e27d5d35e5a39ed7f4d1f662"}
Nov 03 20:30:16.062395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:16.062362    5659 scope.go:122] "RemoveContainer" containerID="858eb317a48e16758deba8c6f355c5d690f65cdd469e62bb2383265ea1b01717"
Nov 03 20:30:16.062683 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:16.062623    5659 scope.go:122] "RemoveContainer" containerID="8fe02ef24f61bd94f9fa933bcdb837e914c88f67e27d5d35e5a39ed7f4d1f662"
Nov 03 20:30:16.063061 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:16.063033    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9_odh-ai-gateway-infra(844f0645-fbd2-4496-9621-343de2fec107)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9" podUID="844f0645-fbd2-4496-9621-343de2fec107"
Nov 03 20:30:16.064978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:16.064920    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272" event={"ID":"a6a06ee0-a5ae-4986-a57d-ec34804041ed","Type":"ContainerStarted","Data":"4b04c2c65e2eed38f52c366dd61de1b7af7a35b838241b7c0c64b9c57d1ce406"}
Nov 03 20:30:16.067238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:16.067212    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4" event={"ID":"75b94007-86d5-47b9-8e7a-e439fd78fb39","Type":"ContainerStarted","Data":"2f778af96939dfffbef9290537dc6bf45c05fa311e3a86a6d6aa3fc8b16988d1"}
Nov 03 20:30:16.071617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:16.071596    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg" event={"ID":"8b99835e-da15-4f01-8df1-da9b74e76ae9","Type":"ContainerStarted","Data":"feb54614eb19f3fd8d142c19ccbbff67d868d8c6fed320f6b08451cc93ff6c25"}
Nov 03 20:30:17.098963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.098927    5659 scope.go:122] "RemoveContainer" containerID="8fe02ef24f61bd94f9fa933bcdb837e914c88f67e27d5d35e5a39ed7f4d1f662"
Nov 03 20:30:17.099660 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:17.099334    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9_odh-ai-gateway-infra(844f0645-fbd2-4496-9621-343de2fec107)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9" podUID="844f0645-fbd2-4496-9621-343de2fec107"
Nov 03 20:30:17.100704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.100680    5659 generic.go:356] "Generic (PLEG): container finished" podID="a6a06ee0-a5ae-4986-a57d-ec34804041ed" containerID="4b04c2c65e2eed38f52c366dd61de1b7af7a35b838241b7c0c64b9c57d1ce406" exitCode=60
Nov 03 20:30:17.100813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.100747    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272" event={"ID":"a6a06ee0-a5ae-4986-a57d-ec34804041ed","Type":"ContainerDied","Data":"4b04c2c65e2eed38f52c366dd61de1b7af7a35b838241b7c0c64b9c57d1ce406"}
Nov 03 20:30:17.100875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.100850    5659 scope.go:122] "RemoveContainer" containerID="e5c91ef4e495c5a0ac02222efcde04497e2735d06e734bff39a23ea40cfd0bf0"
Nov 03 20:30:17.101242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.101222    5659 scope.go:122] "RemoveContainer" containerID="4b04c2c65e2eed38f52c366dd61de1b7af7a35b838241b7c0c64b9c57d1ce406"
Nov 03 20:30:17.101632 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:17.101602    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272_odh-ai-gateway-infra(a6a06ee0-a5ae-4986-a57d-ec34804041ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272" podUID="a6a06ee0-a5ae-4986-a57d-ec34804041ed"
Nov 03 20:30:17.103619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.103595    5659 generic.go:356] "Generic (PLEG): container finished" podID="75b94007-86d5-47b9-8e7a-e439fd78fb39" containerID="2f778af96939dfffbef9290537dc6bf45c05fa311e3a86a6d6aa3fc8b16988d1" exitCode=60
Nov 03 20:30:17.103724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.103664    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4" event={"ID":"75b94007-86d5-47b9-8e7a-e439fd78fb39","Type":"ContainerDied","Data":"2f778af96939dfffbef9290537dc6bf45c05fa311e3a86a6d6aa3fc8b16988d1"}
Nov 03 20:30:17.104029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.104011    5659 scope.go:122] "RemoveContainer" containerID="2f778af96939dfffbef9290537dc6bf45c05fa311e3a86a6d6aa3fc8b16988d1"
Nov 03 20:30:17.104365 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:17.104344    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4_odh-ai-gateway-infra(75b94007-86d5-47b9-8e7a-e439fd78fb39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4" podUID="75b94007-86d5-47b9-8e7a-e439fd78fb39"
Nov 03 20:30:17.106313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.106292    5659 generic.go:356] "Generic (PLEG): container finished" podID="8b99835e-da15-4f01-8df1-da9b74e76ae9" containerID="feb54614eb19f3fd8d142c19ccbbff67d868d8c6fed320f6b08451cc93ff6c25" exitCode=60
Nov 03 20:30:17.106905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.106476    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg" event={"ID":"8b99835e-da15-4f01-8df1-da9b74e76ae9","Type":"ContainerDied","Data":"feb54614eb19f3fd8d142c19ccbbff67d868d8c6fed320f6b08451cc93ff6c25"}
Nov 03 20:30:17.107327 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.107310    5659 scope.go:122] "RemoveContainer" containerID="feb54614eb19f3fd8d142c19ccbbff67d868d8c6fed320f6b08451cc93ff6c25"
Nov 03 20:30:17.107661 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:17.107629    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg_odh-ai-gateway-infra(8b99835e-da15-4f01-8df1-da9b74e76ae9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg" podUID="8b99835e-da15-4f01-8df1-da9b74e76ae9"
Nov 03 20:30:17.109964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.109939    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn" event={"ID":"db088bce-259a-464c-953c-c2fe4d33582a","Type":"ContainerStarted","Data":"699ad638201aee61916df2f3f50152a87847f680b36076e651044612edcfceb8"}
Nov 03 20:30:17.113010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.112984    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"]
Nov 03 20:30:17.118031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.118012    5659 scope.go:122] "RemoveContainer" containerID="03c579c051957fe55ab7e20d8ca5cdb5d512cbd3522e19caf59ab0dd72e0543e"
Nov 03 20:30:17.131914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.131891    5659 scope.go:122] "RemoveContainer" containerID="5bb40208555e5a4ebf3d2d14590aaa03bf1a26d981c693a9c112e776a16ced2b"
Nov 03 20:30:17.155565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.155538    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"]
Nov 03 20:30:17.172428 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:17.172398    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"]
Nov 03 20:30:18.134228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.134188    5659 generic.go:356] "Generic (PLEG): container finished" podID="db088bce-259a-464c-953c-c2fe4d33582a" containerID="699ad638201aee61916df2f3f50152a87847f680b36076e651044612edcfceb8" exitCode=7
Nov 03 20:30:18.134679 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.134236    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn" event={"ID":"db088bce-259a-464c-953c-c2fe4d33582a","Type":"ContainerDied","Data":"699ad638201aee61916df2f3f50152a87847f680b36076e651044612edcfceb8"}
Nov 03 20:30:18.134679 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.134314    5659 scope.go:122] "RemoveContainer" containerID="a2e23e960b39efba7612d13852457148bfc93bf2ad1375c67d0a68d5c6a743bf"
Nov 03 20:30:18.134852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.134832    5659 scope.go:122] "RemoveContainer" containerID="699ad638201aee61916df2f3f50152a87847f680b36076e651044612edcfceb8"
Nov 03 20:30:18.135353 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:18.135325    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn_odh-ai-gateway-infra(db088bce-259a-464c-953c-c2fe4d33582a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn" podUID="db088bce-259a-464c-953c-c2fe4d33582a"
Nov 03 20:30:18.177863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.177268    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"]
Nov 03 20:30:18.484162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.480796    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:18.484162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.481054    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"
Nov 03 20:30:18.529253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.528476    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a06ee0-a5ae-4986-a57d-ec34804041ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a06ee0-a5ae-4986-a57d-ec34804041ed-maas-api-service-ca\") pod \"a6a06ee0-a5ae-4986-a57d-ec34804041ed\" (UID: \"a6a06ee0-a5ae-4986-a57d-ec34804041ed\") "
Nov 03 20:30:18.530252 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.530220    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a06ee0-a5ae-4986-a57d-ec34804041ed-maas-api-service-ca" pod "a6a06ee0-a5ae-4986-a57d-ec34804041ed" (UID: "a6a06ee0-a5ae-4986-a57d-ec34804041ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:18.535020 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.528632    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a06ee0-a5ae-4986-a57d-ec34804041ed-kube-api-access-4ddxf\" (UniqueName: \"kubernetes.io/projected/a6a06ee0-a5ae-4986-a57d-ec34804041ed-kube-api-access-4ddxf\") pod \"a6a06ee0-a5ae-4986-a57d-ec34804041ed\" (UID: \"a6a06ee0-a5ae-4986-a57d-ec34804041ed\") "
Nov 03 20:30:18.537132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.537102    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a06ee0-a5ae-4986-a57d-ec34804041ed-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:18.537468 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.537427    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a06ee0-a5ae-4986-a57d-ec34804041ed-kube-api-access-4ddxf" pod "a6a06ee0-a5ae-4986-a57d-ec34804041ed" (UID: "a6a06ee0-a5ae-4986-a57d-ec34804041ed"). InnerVolumeSpecName "kube-api-access-4ddxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:18.546605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.545582    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"
Nov 03 20:30:18.546605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.546014    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"]
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.547896    5659 cpu_manager.go:414] "containerMap: removing container" podUID="9e384d8e-346d-46fa-9381-f201482e7e8f" containerName="revoke-keys"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.547921    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e384d8e-346d-46fa-9381-f201482e7e8f" containerName="revoke-keys"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.547942    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8b99835e-da15-4f01-8df1-da9b74e76ae9" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.547950    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b99835e-da15-4f01-8df1-da9b74e76ae9" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.547996    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8b99835e-da15-4f01-8df1-da9b74e76ae9" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.548003    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b99835e-da15-4f01-8df1-da9b74e76ae9" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.548057    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a6a06ee0-a5ae-4986-a57d-ec34804041ed" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.548065    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a06ee0-a5ae-4986-a57d-ec34804041ed" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.548083    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a6a06ee0-a5ae-4986-a57d-ec34804041ed" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.548091    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a06ee0-a5ae-4986-a57d-ec34804041ed" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.548104    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a6a06ee0-a5ae-4986-a57d-ec34804041ed" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.548111    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a06ee0-a5ae-4986-a57d-ec34804041ed" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.548294    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a06ee0-a5ae-4986-a57d-ec34804041ed" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.548307    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a06ee0-a5ae-4986-a57d-ec34804041ed" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.548319    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b99835e-da15-4f01-8df1-da9b74e76ae9" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.548336    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b99835e-da15-4f01-8df1-da9b74e76ae9" containerName="cleanup"
Nov 03 20:30:18.548448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.548351    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b99835e-da15-4f01-8df1-da9b74e76ae9" containerName="cleanup"
Nov 03 20:30:18.551823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.551751    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"
Nov 03 20:30:18.555513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.555497    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"]
Nov 03 20:30:18.555645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.555554    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"
Nov 03 20:30:18.592859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.592823    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"]
Nov 03 20:30:18.594253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.594222    5659 cpu_manager.go:414] "containerMap: removing container" podUID="75b94007-86d5-47b9-8e7a-e439fd78fb39" containerName="cleanup"
Nov 03 20:30:18.594253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.594252    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b94007-86d5-47b9-8e7a-e439fd78fb39" containerName="cleanup"
Nov 03 20:30:18.594458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.594287    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8b99835e-da15-4f01-8df1-da9b74e76ae9" containerName="cleanup"
Nov 03 20:30:18.594458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.594294    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b99835e-da15-4f01-8df1-da9b74e76ae9" containerName="cleanup"
Nov 03 20:30:18.594458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.594313    5659 cpu_manager.go:414] "containerMap: removing container" podUID="75b94007-86d5-47b9-8e7a-e439fd78fb39" containerName="cleanup"
Nov 03 20:30:18.594458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.594320    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b94007-86d5-47b9-8e7a-e439fd78fb39" containerName="cleanup"
Nov 03 20:30:18.594718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.594567    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b94007-86d5-47b9-8e7a-e439fd78fb39" containerName="cleanup"
Nov 03 20:30:18.594718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.594587    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b94007-86d5-47b9-8e7a-e439fd78fb39" containerName="cleanup"
Nov 03 20:30:18.594718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.594601    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b94007-86d5-47b9-8e7a-e439fd78fb39" containerName="cleanup"
Nov 03 20:30:18.594718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.594616    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a06ee0-a5ae-4986-a57d-ec34804041ed" containerName="cleanup"
Nov 03 20:30:18.597596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.597575    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"
Nov 03 20:30:18.603527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.603086    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:30:18.608569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.608550    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"]
Nov 03 20:30:18.627599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.626299    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.637752    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b94007-86d5-47b9-8e7a-e439fd78fb39-kube-api-access-v4zdg\" (UniqueName: \"kubernetes.io/projected/75b94007-86d5-47b9-8e7a-e439fd78fb39-kube-api-access-v4zdg\") pod \"75b94007-86d5-47b9-8e7a-e439fd78fb39\" (UID: \"75b94007-86d5-47b9-8e7a-e439fd78fb39\") "
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.637930    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b94007-86d5-47b9-8e7a-e439fd78fb39-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b94007-86d5-47b9-8e7a-e439fd78fb39-maas-api-service-ca\") pod \"75b94007-86d5-47b9-8e7a-e439fd78fb39\" (UID: \"75b94007-86d5-47b9-8e7a-e439fd78fb39\") "
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.638026    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b99835e-da15-4f01-8df1-da9b74e76ae9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b99835e-da15-4f01-8df1-da9b74e76ae9-maas-api-service-ca\") pod \"8b99835e-da15-4f01-8df1-da9b74e76ae9\" (UID: \"8b99835e-da15-4f01-8df1-da9b74e76ae9\") "
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.638328    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b94007-86d5-47b9-8e7a-e439fd78fb39-maas-api-service-ca" pod "75b94007-86d5-47b9-8e7a-e439fd78fb39" (UID: "75b94007-86d5-47b9-8e7a-e439fd78fb39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.638382    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b99835e-da15-4f01-8df1-da9b74e76ae9-maas-api-service-ca" pod "8b99835e-da15-4f01-8df1-da9b74e76ae9" (UID: "8b99835e-da15-4f01-8df1-da9b74e76ae9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.638722    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b99835e-da15-4f01-8df1-da9b74e76ae9-kube-api-access-jzn4q\" (UniqueName: \"kubernetes.io/projected/8b99835e-da15-4f01-8df1-da9b74e76ae9-kube-api-access-jzn4q\") pod \"8b99835e-da15-4f01-8df1-da9b74e76ae9\" (UID: \"8b99835e-da15-4f01-8df1-da9b74e76ae9\") "
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.639105    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-q8zvp\" (UID: \"f4069450-679d-44cb-ab4e-44c7fea95e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.639200    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde691e8-b602-449e-a763-a94adcb073c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg\" (UID: \"bde691e8-b602-449e-a763-a94adcb073c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.639345    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggcrd\" (UniqueName: \"kubernetes.io/projected/bde691e8-b602-449e-a763-a94adcb073c5-kube-api-access-ggcrd\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg\" (UID: \"bde691e8-b602-449e-a763-a94adcb073c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.639458    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfpq2\" (UniqueName: \"kubernetes.io/projected/f4069450-679d-44cb-ab4e-44c7fea95e76-kube-api-access-jfpq2\") pod \"maas-api-revoke-sub-5bac6f1e-q8zvp\" (UID: \"f4069450-679d-44cb-ab4e-44c7fea95e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.639721    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-q8zvp\" (UID: \"f4069450-679d-44cb-ab4e-44c7fea95e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.639877    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ddxf\" (UniqueName: \"kubernetes.io/projected/a6a06ee0-a5ae-4986-a57d-ec34804041ed-kube-api-access-4ddxf\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.639896    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b94007-86d5-47b9-8e7a-e439fd78fb39-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:18.640512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.639911    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b99835e-da15-4f01-8df1-da9b74e76ae9-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:18.644211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.642111    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"]
Nov 03 20:30:18.644211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.643503    5659 cpu_manager.go:414] "containerMap: removing container" podUID="75b94007-86d5-47b9-8e7a-e439fd78fb39" containerName="cleanup"
Nov 03 20:30:18.644211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.643523    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b94007-86d5-47b9-8e7a-e439fd78fb39" containerName="cleanup"
Nov 03 20:30:18.645022 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.644609    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b99835e-da15-4f01-8df1-da9b74e76ae9-kube-api-access-jzn4q" pod "8b99835e-da15-4f01-8df1-da9b74e76ae9" (UID: "8b99835e-da15-4f01-8df1-da9b74e76ae9"). InnerVolumeSpecName "kube-api-access-jzn4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:18.669356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.669308    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b94007-86d5-47b9-8e7a-e439fd78fb39-kube-api-access-v4zdg" pod "75b94007-86d5-47b9-8e7a-e439fd78fb39" (UID: "75b94007-86d5-47b9-8e7a-e439fd78fb39"). InnerVolumeSpecName "kube-api-access-v4zdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:18.670967 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.670904    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"
Nov 03 20:30:18.693112 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.693078    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"]
Nov 03 20:30:18.741818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.741724    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-x4nx4\" (UID: \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"
Nov 03 20:30:18.741973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.741832    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rzmd\" (UniqueName: \"kubernetes.io/projected/8d2e7d4e-68f3-48a9-9fff-133223dadb65-kube-api-access-6rzmd\") pod \"maas-api-revoke-sub-403cf98d-x4nx4\" (UID: \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"
Nov 03 20:30:18.742018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.741972    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-q8zvp\" (UID: \"f4069450-679d-44cb-ab4e-44c7fea95e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"
Nov 03 20:30:18.742164 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.742147    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-x4nx4\" (UID: \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"
Nov 03 20:30:18.742203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.742193    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-q8zvp\" (UID: \"f4069450-679d-44cb-ab4e-44c7fea95e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"
Nov 03 20:30:18.742251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.742236    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde691e8-b602-449e-a763-a94adcb073c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg\" (UID: \"bde691e8-b602-449e-a763-a94adcb073c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"
Nov 03 20:30:18.742464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.742436    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggcrd\" (UniqueName: \"kubernetes.io/projected/bde691e8-b602-449e-a763-a94adcb073c5-kube-api-access-ggcrd\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg\" (UID: \"bde691e8-b602-449e-a763-a94adcb073c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"
Nov 03 20:30:18.742752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.742732    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfpq2\" (UniqueName: \"kubernetes.io/projected/f4069450-679d-44cb-ab4e-44c7fea95e76-kube-api-access-jfpq2\") pod \"maas-api-revoke-sub-5bac6f1e-q8zvp\" (UID: \"f4069450-679d-44cb-ab4e-44c7fea95e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"
Nov 03 20:30:18.743006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.742981    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-q8zvp\" (UID: \"f4069450-679d-44cb-ab4e-44c7fea95e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"
Nov 03 20:30:18.743171 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.743067    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzn4q\" (UniqueName: \"kubernetes.io/projected/8b99835e-da15-4f01-8df1-da9b74e76ae9-kube-api-access-jzn4q\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:18.743274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.743183    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4zdg\" (UniqueName: \"kubernetes.io/projected/75b94007-86d5-47b9-8e7a-e439fd78fb39-kube-api-access-v4zdg\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:18.743274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.743068    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde691e8-b602-449e-a763-a94adcb073c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg\" (UID: \"bde691e8-b602-449e-a763-a94adcb073c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"
Nov 03 20:30:18.744452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.744432    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-q8zvp\" (UID: \"f4069450-679d-44cb-ab4e-44c7fea95e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"
Nov 03 20:30:18.756669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.756642    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggcrd\" (UniqueName: \"kubernetes.io/projected/bde691e8-b602-449e-a763-a94adcb073c5-kube-api-access-ggcrd\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg\" (UID: \"bde691e8-b602-449e-a763-a94adcb073c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"
Nov 03 20:30:18.756874 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.756854    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfpq2\" (UniqueName: \"kubernetes.io/projected/f4069450-679d-44cb-ab4e-44c7fea95e76-kube-api-access-jfpq2\") pod \"maas-api-revoke-sub-5bac6f1e-q8zvp\" (UID: \"f4069450-679d-44cb-ab4e-44c7fea95e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"
Nov 03 20:30:18.845145 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.845106    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-x4nx4\" (UID: \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"
Nov 03 20:30:18.845145 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.845154    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rzmd\" (UniqueName: \"kubernetes.io/projected/8d2e7d4e-68f3-48a9-9fff-133223dadb65-kube-api-access-6rzmd\") pod \"maas-api-revoke-sub-403cf98d-x4nx4\" (UID: \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"
Nov 03 20:30:18.845434 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.845305    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-x4nx4\" (UID: \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"
Nov 03 20:30:18.846210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.846182    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-x4nx4\" (UID: \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"
Nov 03 20:30:18.847583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.847555    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-x4nx4\" (UID: \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"
Nov 03 20:30:18.857477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.857450    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rzmd\" (UniqueName: \"kubernetes.io/projected/8d2e7d4e-68f3-48a9-9fff-133223dadb65-kube-api-access-6rzmd\") pod \"maas-api-revoke-sub-403cf98d-x4nx4\" (UID: \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"
Nov 03 20:30:18.875905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.875863    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"
Nov 03 20:30:18.923569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:18.923530    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"
Nov 03 20:30:19.006258 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.006162    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"
Nov 03 20:30:19.128429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.128392    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:19.184504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.183883    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"
Nov 03 20:30:19.192832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.192748    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272" event={"ID":"a6a06ee0-a5ae-4986-a57d-ec34804041ed","Type":"ContainerDied","Data":"9810ab93039ef006c832e7a490fe25b1d88ad9bb1fa2ebe64d23a3ba89435ad9"}
Nov 03 20:30:19.192832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.192835    5659 scope.go:122] "RemoveContainer" containerID="4b04c2c65e2eed38f52c366dd61de1b7af7a35b838241b7c0c64b9c57d1ce406"
Nov 03 20:30:19.202027 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:19.201986    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde691e8_b602_449e_a763_a94adcb073c5.slice/crio-d7f40a42c320a273f424a83d7fc4b50b8304c02ee2576b8d6279c216c64d15cb WatchSource:0}: Error finding container d7f40a42c320a273f424a83d7fc4b50b8304c02ee2576b8d6279c216c64d15cb: Status 404 returned error can't find the container with id d7f40a42c320a273f424a83d7fc4b50b8304c02ee2576b8d6279c216c64d15cb
Nov 03 20:30:19.211532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.207173    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4" event={"ID":"75b94007-86d5-47b9-8e7a-e439fd78fb39","Type":"ContainerDied","Data":"baac2bc193b38611cfef13d8f2ef88580aa457da6e030d56f5b4cdb529a5cd59"}
Nov 03 20:30:19.211532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.207335    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"
Nov 03 20:30:19.228198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.225292    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg" event={"ID":"8b99835e-da15-4f01-8df1-da9b74e76ae9","Type":"ContainerDied","Data":"0cb52c55257b19752f7124fe8d62f7c633a7883026b11ace0c1e0fb543014a41"}
Nov 03 20:30:19.228198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.225454    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"
Nov 03 20:30:19.283571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.283543    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"]
Nov 03 20:30:19.301412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.301199    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"]
Nov 03 20:30:19.302718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.302693    5659 scope.go:122] "RemoveContainer" containerID="2f778af96939dfffbef9290537dc6bf45c05fa311e3a86a6d6aa3fc8b16988d1"
Nov 03 20:30:19.316471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.316434    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"]
Nov 03 20:30:19.320944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.320921    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-660d95-29895630-sk6gg"]
Nov 03 20:30:19.333636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.333326    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"]
Nov 03 20:30:19.333636 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:19.333590    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2e7d4e_68f3_48a9_9fff_133223dadb65.slice/crio-75e9ebb9847757987f0508032a07734efa3554d8543c7fb727b69e53f3f40ca6 WatchSource:0}: Error finding container 75e9ebb9847757987f0508032a07734efa3554d8543c7fb727b69e53f3f40ca6: Status 404 returned error can't find the container with id 75e9ebb9847757987f0508032a07734efa3554d8543c7fb727b69e53f3f40ca6
Nov 03 20:30:19.342732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.342196    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-eb69cb-29895630-4b2t4"]
Nov 03 20:30:19.357335 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.354391    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"]
Nov 03 20:30:19.358625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.358602    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"]
Nov 03 20:30:19.368457 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.368287    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-942930-29895630-gc272"]
Nov 03 20:30:19.442758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.442729    5659 scope.go:122] "RemoveContainer" containerID="feb54614eb19f3fd8d142c19ccbbff67d868d8c6fed320f6b08451cc93ff6c25"
Nov 03 20:30:19.521466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.521401    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"
Nov 03 20:30:19.553289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.553254    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"]
Nov 03 20:30:19.554155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.554138    5659 cpu_manager.go:414] "containerMap: removing container" podUID="db088bce-259a-464c-953c-c2fe4d33582a" containerName="cleanup"
Nov 03 20:30:19.554218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.554156    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db088bce-259a-464c-953c-c2fe4d33582a" containerName="cleanup"
Nov 03 20:30:19.554218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.554204    5659 cpu_manager.go:414] "containerMap: removing container" podUID="db088bce-259a-464c-953c-c2fe4d33582a" containerName="cleanup"
Nov 03 20:30:19.554218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.554209    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db088bce-259a-464c-953c-c2fe4d33582a" containerName="cleanup"
Nov 03 20:30:19.554342 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.554332    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="db088bce-259a-464c-953c-c2fe4d33582a" containerName="cleanup"
Nov 03 20:30:19.554376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.554343    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="db088bce-259a-464c-953c-c2fe4d33582a" containerName="cleanup"
Nov 03 20:30:19.557955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.557933    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"
Nov 03 20:30:19.568129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.568094    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"]
Nov 03 20:30:19.571019 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.570987    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db088bce-259a-464c-953c-c2fe4d33582a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db088bce-259a-464c-953c-c2fe4d33582a-maas-api-service-ca\") pod \"db088bce-259a-464c-953c-c2fe4d33582a\" (UID: \"db088bce-259a-464c-953c-c2fe4d33582a\") "
Nov 03 20:30:19.571188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.571088    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db088bce-259a-464c-953c-c2fe4d33582a-kube-api-access-8t79q\" (UniqueName: \"kubernetes.io/projected/db088bce-259a-464c-953c-c2fe4d33582a-kube-api-access-8t79q\") pod \"db088bce-259a-464c-953c-c2fe4d33582a\" (UID: \"db088bce-259a-464c-953c-c2fe4d33582a\") "
Nov 03 20:30:19.582706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.571538    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db088bce-259a-464c-953c-c2fe4d33582a-maas-api-service-ca" pod "db088bce-259a-464c-953c-c2fe4d33582a" (UID: "db088bce-259a-464c-953c-c2fe4d33582a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:19.584708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.584677    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db088bce-259a-464c-953c-c2fe4d33582a-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:19.639855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.639812    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db088bce-259a-464c-953c-c2fe4d33582a-kube-api-access-8t79q" pod "db088bce-259a-464c-953c-c2fe4d33582a" (UID: "db088bce-259a-464c-953c-c2fe4d33582a"). InnerVolumeSpecName "kube-api-access-8t79q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:19.686253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.686209    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qnm\" (UniqueName: \"kubernetes.io/projected/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-kube-api-access-m5qnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf\" (UID: \"24fc2329-33a9-4d19-a4ca-a1b0b83c893c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"
Nov 03 20:30:19.686456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.686373    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf\" (UID: \"24fc2329-33a9-4d19-a4ca-a1b0b83c893c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"
Nov 03 20:30:19.686666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.686650    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t79q\" (UniqueName: \"kubernetes.io/projected/db088bce-259a-464c-953c-c2fe4d33582a-kube-api-access-8t79q\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:19.788726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.788634    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qnm\" (UniqueName: \"kubernetes.io/projected/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-kube-api-access-m5qnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf\" (UID: \"24fc2329-33a9-4d19-a4ca-a1b0b83c893c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"
Nov 03 20:30:19.788912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.788808    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf\" (UID: \"24fc2329-33a9-4d19-a4ca-a1b0b83c893c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"
Nov 03 20:30:19.789741 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.789694    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf\" (UID: \"24fc2329-33a9-4d19-a4ca-a1b0b83c893c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"
Nov 03 20:30:19.832687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.832653    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qnm\" (UniqueName: \"kubernetes.io/projected/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-kube-api-access-m5qnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf\" (UID: \"24fc2329-33a9-4d19-a4ca-a1b0b83c893c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"
Nov 03 20:30:19.886766 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.886726    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"
Nov 03 20:30:19.964656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:19.964607    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:20.021945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.018205    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b94007-86d5-47b9-8e7a-e439fd78fb39" path="/var/lib/kubelet/pods/75b94007-86d5-47b9-8e7a-e439fd78fb39/volumes"
Nov 03 20:30:20.021945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.019162    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b99835e-da15-4f01-8df1-da9b74e76ae9" path="/var/lib/kubelet/pods/8b99835e-da15-4f01-8df1-da9b74e76ae9/volumes"
Nov 03 20:30:20.021945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.019856    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a06ee0-a5ae-4986-a57d-ec34804041ed" path="/var/lib/kubelet/pods/a6a06ee0-a5ae-4986-a57d-ec34804041ed/volumes"
Nov 03 20:30:20.252236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.252199    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp" event={"ID":"f4069450-679d-44cb-ab4e-44c7fea95e76","Type":"ContainerStarted","Data":"d08b598e6e130767ef7b4ee8e38a7f83d5b9dabdc1e62537e72b3cdb1691ccda"}
Nov 03 20:30:20.252684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.252259    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp" event={"ID":"f4069450-679d-44cb-ab4e-44c7fea95e76","Type":"ContainerStarted","Data":"9bbe1ce0057512fdb828cfc4ecfded07448a1092225c2dacff418cb01b404289"}
Nov 03 20:30:20.256130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.256101    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4" event={"ID":"8d2e7d4e-68f3-48a9-9fff-133223dadb65","Type":"ContainerStarted","Data":"b06ac3de6a1648f1401a0601a9eb9e9071f6e3dea0a5be5056bbc7cf0d081afb"}
Nov 03 20:30:20.256302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.256166    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4" event={"ID":"8d2e7d4e-68f3-48a9-9fff-133223dadb65","Type":"ContainerStarted","Data":"75e9ebb9847757987f0508032a07734efa3554d8543c7fb727b69e53f3f40ca6"}
Nov 03 20:30:20.258443 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.258422    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"
Nov 03 20:30:20.258443 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.258431    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn" event={"ID":"db088bce-259a-464c-953c-c2fe4d33582a","Type":"ContainerDied","Data":"c2a7edd3ab82c5354682d7b6506bf39830d86fb73bdca6086f749a79e4268cba"}
Nov 03 20:30:20.258627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.258477    5659 scope.go:122] "RemoveContainer" containerID="699ad638201aee61916df2f3f50152a87847f680b36076e651044612edcfceb8"
Nov 03 20:30:20.260398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.260350    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg" event={"ID":"bde691e8-b602-449e-a763-a94adcb073c5","Type":"ContainerStarted","Data":"06615cbc99ee6e64fd10b73d0ef26b1e0e5544e4f09f56a7b34c0a1c7cf56383"}
Nov 03 20:30:20.260537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.260438    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg" event={"ID":"bde691e8-b602-449e-a763-a94adcb073c5","Type":"ContainerStarted","Data":"d7f40a42c320a273f424a83d7fc4b50b8304c02ee2576b8d6279c216c64d15cb"}
Nov 03 20:30:20.279381 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.279334    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp" podStartSLOduration=8.279319614 podStartE2EDuration="8.279319614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:20.278266415 +0000 UTC m=+3172.433168830" watchObservedRunningTime="2026-11-03 20:30:20.279319614 +0000 UTC m=+3172.434222032"
Nov 03 20:30:20.305542 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:20.305463    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fc2329_33a9_4d19_a4ca_a1b0b83c893c.slice/crio-ea52c1e1ad6f54c5d632dc5e1dc178a76e625ce76c30c7fdbe3a519185529233 WatchSource:0}: Error finding container ea52c1e1ad6f54c5d632dc5e1dc178a76e625ce76c30c7fdbe3a519185529233: Status 404 returned error can't find the container with id ea52c1e1ad6f54c5d632dc5e1dc178a76e625ce76c30c7fdbe3a519185529233
Nov 03 20:30:20.305992 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.305951    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"]
Nov 03 20:30:20.322103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.322038    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg" podStartSLOduration=20.322020006 podStartE2EDuration="20.322020006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:20.320826604 +0000 UTC m=+3172.475729023" watchObservedRunningTime="2026-11-03 20:30:20.322020006 +0000 UTC m=+3172.476922426"
Nov 03 20:30:20.354452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.354384    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4" podStartSLOduration=5.354363283 podStartE2EDuration="5.354363283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:20.342923456 +0000 UTC m=+3172.497825871" watchObservedRunningTime="2026-11-03 20:30:20.354363283 +0000 UTC m=+3172.509265703"
Nov 03 20:30:20.372586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.372550    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"]
Nov 03 20:30:20.376640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:20.376609    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7b70d-29895630-m6zjn"]
Nov 03 20:30:21.588031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:21.587985    5659 generic.go:356] "Generic (PLEG): container finished" podID="bde691e8-b602-449e-a763-a94adcb073c5" containerID="06615cbc99ee6e64fd10b73d0ef26b1e0e5544e4f09f56a7b34c0a1c7cf56383" exitCode=60
Nov 03 20:30:21.623805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:21.623758    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg" event={"ID":"bde691e8-b602-449e-a763-a94adcb073c5","Type":"ContainerDied","Data":"06615cbc99ee6e64fd10b73d0ef26b1e0e5544e4f09f56a7b34c0a1c7cf56383"}
Nov 03 20:30:21.663504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:21.650356    5659 scope.go:122] "RemoveContainer" containerID="06615cbc99ee6e64fd10b73d0ef26b1e0e5544e4f09f56a7b34c0a1c7cf56383"
Nov 03 20:30:21.744507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:21.733825    5659 generic.go:356] "Generic (PLEG): container finished" podID="f4069450-679d-44cb-ab4e-44c7fea95e76" containerID="d08b598e6e130767ef7b4ee8e38a7f83d5b9dabdc1e62537e72b3cdb1691ccda" exitCode=22
Nov 03 20:30:21.744507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:21.734077    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp" event={"ID":"f4069450-679d-44cb-ab4e-44c7fea95e76","Type":"ContainerDied","Data":"d08b598e6e130767ef7b4ee8e38a7f83d5b9dabdc1e62537e72b3cdb1691ccda"}
Nov 03 20:30:21.744507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:21.734793    5659 scope.go:122] "RemoveContainer" containerID="d08b598e6e130767ef7b4ee8e38a7f83d5b9dabdc1e62537e72b3cdb1691ccda"
Nov 03 20:30:21.952870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:21.931371    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:21.952870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:21.939755    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf" event={"ID":"24fc2329-33a9-4d19-a4ca-a1b0b83c893c","Type":"ContainerStarted","Data":"77fd94cd388320f5de1afb2b1d17f9be5f9b5acbdf4bbfea4a7470f0f636a53d"}
Nov 03 20:30:21.952870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:21.939828    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf" event={"ID":"24fc2329-33a9-4d19-a4ca-a1b0b83c893c","Type":"ContainerStarted","Data":"ea52c1e1ad6f54c5d632dc5e1dc178a76e625ce76c30c7fdbe3a519185529233"}
Nov 03 20:30:21.991936 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:21.991889    5659 scope.go:122] "RemoveContainer" containerID="77fd94cd388320f5de1afb2b1d17f9be5f9b5acbdf4bbfea4a7470f0f636a53d"
Nov 03 20:30:22.111159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:22.111124    5659 generic.go:356] "Generic (PLEG): container finished" podID="8d2e7d4e-68f3-48a9-9fff-133223dadb65" containerID="b06ac3de6a1648f1401a0601a9eb9e9071f6e3dea0a5be5056bbc7cf0d081afb" exitCode=22
Nov 03 20:30:22.111737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:22.111716    5659 scope.go:122] "RemoveContainer" containerID="b06ac3de6a1648f1401a0601a9eb9e9071f6e3dea0a5be5056bbc7cf0d081afb"
Nov 03 20:30:22.181103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:22.181066    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db088bce-259a-464c-953c-c2fe4d33582a" path="/var/lib/kubelet/pods/db088bce-259a-464c-953c-c2fe4d33582a/volumes"
Nov 03 20:30:22.182205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:22.181994    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4" event={"ID":"8d2e7d4e-68f3-48a9-9fff-133223dadb65","Type":"ContainerDied","Data":"b06ac3de6a1648f1401a0601a9eb9e9071f6e3dea0a5be5056bbc7cf0d081afb"}
Nov 03 20:30:23.122358 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:23.122264    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp" event={"ID":"f4069450-679d-44cb-ab4e-44c7fea95e76","Type":"ContainerStarted","Data":"8e81a8a8d43b9f09678b566342adb15ac116e3850732a7d4d2509ebd1c948dfa"}
Nov 03 20:30:23.124413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:23.124382    5659 generic.go:356] "Generic (PLEG): container finished" podID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c" containerID="77fd94cd388320f5de1afb2b1d17f9be5f9b5acbdf4bbfea4a7470f0f636a53d" exitCode=60
Nov 03 20:30:23.124607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:23.124431    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf" event={"ID":"24fc2329-33a9-4d19-a4ca-a1b0b83c893c","Type":"ContainerDied","Data":"77fd94cd388320f5de1afb2b1d17f9be5f9b5acbdf4bbfea4a7470f0f636a53d"}
Nov 03 20:30:23.124607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:23.124504    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf" event={"ID":"24fc2329-33a9-4d19-a4ca-a1b0b83c893c","Type":"ContainerStarted","Data":"710024563df86faadfe631844747b9db72429ec46d281f27ede7174186bc0d83"}
Nov 03 20:30:23.126653 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:23.126631    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4" event={"ID":"8d2e7d4e-68f3-48a9-9fff-133223dadb65","Type":"ContainerStarted","Data":"ef18ba699ab19799d1dbf82cd4f6942b414280110c39d9f4b9f3dfab048479f4"}
Nov 03 20:30:23.128640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:23.128619    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg" event={"ID":"bde691e8-b602-449e-a763-a94adcb073c5","Type":"ContainerStarted","Data":"36d5655913b862378ff0c337314951b25c5f3977d696774bf3f68e148fc58df6"}
Nov 03 20:30:23.220892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:23.220835    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf" podStartSLOduration=23.220820063 podStartE2EDuration="23.220820063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:23.220076722 +0000 UTC m=+3175.374979142" watchObservedRunningTime="2026-11-03 20:30:23.220820063 +0000 UTC m=+3175.375722482"
Nov 03 20:30:23.381555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:23.379125    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:24.153506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.149276    5659 generic.go:356] "Generic (PLEG): container finished" podID="8d2e7d4e-68f3-48a9-9fff-133223dadb65" containerID="ef18ba699ab19799d1dbf82cd4f6942b414280110c39d9f4b9f3dfab048479f4" exitCode=22
Nov 03 20:30:24.153506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.149411    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4" event={"ID":"8d2e7d4e-68f3-48a9-9fff-133223dadb65","Type":"ContainerDied","Data":"ef18ba699ab19799d1dbf82cd4f6942b414280110c39d9f4b9f3dfab048479f4"}
Nov 03 20:30:24.153506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.149475    5659 scope.go:122] "RemoveContainer" containerID="b06ac3de6a1648f1401a0601a9eb9e9071f6e3dea0a5be5056bbc7cf0d081afb"
Nov 03 20:30:24.153506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.149890    5659 scope.go:122] "RemoveContainer" containerID="ef18ba699ab19799d1dbf82cd4f6942b414280110c39d9f4b9f3dfab048479f4"
Nov 03 20:30:24.153506 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:24.150279    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-x4nx4_odh-ai-gateway-infra(8d2e7d4e-68f3-48a9-9fff-133223dadb65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4" podUID="8d2e7d4e-68f3-48a9-9fff-133223dadb65"
Nov 03 20:30:24.153506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.152088    5659 generic.go:356] "Generic (PLEG): container finished" podID="bde691e8-b602-449e-a763-a94adcb073c5" containerID="36d5655913b862378ff0c337314951b25c5f3977d696774bf3f68e148fc58df6" exitCode=60
Nov 03 20:30:24.153506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.152226    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg" event={"ID":"bde691e8-b602-449e-a763-a94adcb073c5","Type":"ContainerDied","Data":"36d5655913b862378ff0c337314951b25c5f3977d696774bf3f68e148fc58df6"}
Nov 03 20:30:24.153506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.152658    5659 scope.go:122] "RemoveContainer" containerID="36d5655913b862378ff0c337314951b25c5f3977d696774bf3f68e148fc58df6"
Nov 03 20:30:24.153506 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:24.153024    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg_odh-ai-gateway-infra(bde691e8-b602-449e-a763-a94adcb073c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg" podUID="bde691e8-b602-449e-a763-a94adcb073c5"
Nov 03 20:30:24.160499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.155682    5659 generic.go:356] "Generic (PLEG): container finished" podID="f4069450-679d-44cb-ab4e-44c7fea95e76" containerID="8e81a8a8d43b9f09678b566342adb15ac116e3850732a7d4d2509ebd1c948dfa" exitCode=22
Nov 03 20:30:24.160499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.155911    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp" event={"ID":"f4069450-679d-44cb-ab4e-44c7fea95e76","Type":"ContainerDied","Data":"8e81a8a8d43b9f09678b566342adb15ac116e3850732a7d4d2509ebd1c948dfa"}
Nov 03 20:30:24.160499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.156955    5659 scope.go:122] "RemoveContainer" containerID="8e81a8a8d43b9f09678b566342adb15ac116e3850732a7d4d2509ebd1c948dfa"
Nov 03 20:30:24.160499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.159329    5659 generic.go:356] "Generic (PLEG): container finished" podID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c" containerID="710024563df86faadfe631844747b9db72429ec46d281f27ede7174186bc0d83" exitCode=60
Nov 03 20:30:24.160499 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:24.159406    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-q8zvp_odh-ai-gateway-infra(f4069450-679d-44cb-ab4e-44c7fea95e76)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp" podUID="f4069450-679d-44cb-ab4e-44c7fea95e76"
Nov 03 20:30:24.160499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.159453    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf" event={"ID":"24fc2329-33a9-4d19-a4ca-a1b0b83c893c","Type":"ContainerDied","Data":"710024563df86faadfe631844747b9db72429ec46d281f27ede7174186bc0d83"}
Nov 03 20:30:24.160499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.159813    5659 scope.go:122] "RemoveContainer" containerID="710024563df86faadfe631844747b9db72429ec46d281f27ede7174186bc0d83"
Nov 03 20:30:24.160499 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:24.160114    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf_odh-ai-gateway-infra(24fc2329-33a9-4d19-a4ca-a1b0b83c893c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf" podUID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c"
Nov 03 20:30:24.167039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.167018    5659 scope.go:122] "RemoveContainer" containerID="06615cbc99ee6e64fd10b73d0ef26b1e0e5544e4f09f56a7b34c0a1c7cf56383"
Nov 03 20:30:24.186364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.186334    5659 scope.go:122] "RemoveContainer" containerID="d08b598e6e130767ef7b4ee8e38a7f83d5b9dabdc1e62537e72b3cdb1691ccda"
Nov 03 20:30:24.204424 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.204400    5659 scope.go:122] "RemoveContainer" containerID="77fd94cd388320f5de1afb2b1d17f9be5f9b5acbdf4bbfea4a7470f0f636a53d"
Nov 03 20:30:24.766506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.743842    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:24.962208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:24.962169    5659 scope.go:122] "RemoveContainer" containerID="8602769d5612cc2c0f2a528ee4b64f30e3a87b9148fe1837fd414b8244353d53"
Nov 03 20:30:25.176302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:25.176261    5659 scope.go:122] "RemoveContainer" containerID="ef18ba699ab19799d1dbf82cd4f6942b414280110c39d9f4b9f3dfab048479f4"
Nov 03 20:30:25.179160 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:25.177055    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-x4nx4_odh-ai-gateway-infra(8d2e7d4e-68f3-48a9-9fff-133223dadb65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4" podUID="8d2e7d4e-68f3-48a9-9fff-133223dadb65"
Nov 03 20:30:25.179160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:25.178941    5659 scope.go:122] "RemoveContainer" containerID="36d5655913b862378ff0c337314951b25c5f3977d696774bf3f68e148fc58df6"
Nov 03 20:30:25.179762 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:25.179719    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg_odh-ai-gateway-infra(bde691e8-b602-449e-a763-a94adcb073c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg" podUID="bde691e8-b602-449e-a763-a94adcb073c5"
Nov 03 20:30:25.181755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:25.181730    5659 scope.go:122] "RemoveContainer" containerID="8e81a8a8d43b9f09678b566342adb15ac116e3850732a7d4d2509ebd1c948dfa"
Nov 03 20:30:25.182300 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:25.182271    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-q8zvp_odh-ai-gateway-infra(f4069450-679d-44cb-ab4e-44c7fea95e76)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp" podUID="f4069450-679d-44cb-ab4e-44c7fea95e76"
Nov 03 20:30:25.183884 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:25.183847    5659 scope.go:122] "RemoveContainer" containerID="710024563df86faadfe631844747b9db72429ec46d281f27ede7174186bc0d83"
Nov 03 20:30:25.184520 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:25.184477    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf_odh-ai-gateway-infra(24fc2329-33a9-4d19-a4ca-a1b0b83c893c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf" podUID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c"
Nov 03 20:30:25.291602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:25.291548    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:25.467008 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:25.466968    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:26.145445 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:26.145353    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:26.263603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:26.263308    5659 generic.go:356] "Generic (PLEG): container finished" podID="659d88c4-b3c0-49d1-b799-7c746abdef13" containerID="4757e4763bddcf6e6571e0950d5d84d38cbe9260bb810ff837475d0e0eb57b14" exitCode=60
Nov 03 20:30:26.263603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:26.263520    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp" event={"ID":"659d88c4-b3c0-49d1-b799-7c746abdef13","Type":"ContainerDied","Data":"4757e4763bddcf6e6571e0950d5d84d38cbe9260bb810ff837475d0e0eb57b14"}
Nov 03 20:30:26.263603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:26.263585    5659 scope.go:122] "RemoveContainer" containerID="8602769d5612cc2c0f2a528ee4b64f30e3a87b9148fe1837fd414b8244353d53"
Nov 03 20:30:26.264401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:26.264140    5659 scope.go:122] "RemoveContainer" containerID="4757e4763bddcf6e6571e0950d5d84d38cbe9260bb810ff837475d0e0eb57b14"
Nov 03 20:30:26.264678 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:26.264645    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp_odh-ai-gateway-infra(659d88c4-b3c0-49d1-b799-7c746abdef13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp" podUID="659d88c4-b3c0-49d1-b799-7c746abdef13"
Nov 03 20:30:26.926687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:26.926653    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:27.295936 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.295900    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"]
Nov 03 20:30:27.499658 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.499630    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"
Nov 03 20:30:27.524782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.524751    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659d88c4-b3c0-49d1-b799-7c746abdef13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d88c4-b3c0-49d1-b799-7c746abdef13-maas-api-service-ca\") pod \"659d88c4-b3c0-49d1-b799-7c746abdef13\" (UID: \"659d88c4-b3c0-49d1-b799-7c746abdef13\") "
Nov 03 20:30:27.524965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.524816    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659d88c4-b3c0-49d1-b799-7c746abdef13-kube-api-access-xn5pk\" (UniqueName: \"kubernetes.io/projected/659d88c4-b3c0-49d1-b799-7c746abdef13-kube-api-access-xn5pk\") pod \"659d88c4-b3c0-49d1-b799-7c746abdef13\" (UID: \"659d88c4-b3c0-49d1-b799-7c746abdef13\") "
Nov 03 20:30:27.525187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.525159    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659d88c4-b3c0-49d1-b799-7c746abdef13-maas-api-service-ca" pod "659d88c4-b3c0-49d1-b799-7c746abdef13" (UID: "659d88c4-b3c0-49d1-b799-7c746abdef13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:27.525643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.525618    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659d88c4-b3c0-49d1-b799-7c746abdef13-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:27.527281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.527255    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659d88c4-b3c0-49d1-b799-7c746abdef13-kube-api-access-xn5pk" pod "659d88c4-b3c0-49d1-b799-7c746abdef13" (UID: "659d88c4-b3c0-49d1-b799-7c746abdef13"). InnerVolumeSpecName "kube-api-access-xn5pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:27.548062 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.548028    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"]
Nov 03 20:30:27.551377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.549518    5659 cpu_manager.go:414] "containerMap: removing container" podUID="659d88c4-b3c0-49d1-b799-7c746abdef13" containerName="cleanup"
Nov 03 20:30:27.551377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.549543    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659d88c4-b3c0-49d1-b799-7c746abdef13" containerName="cleanup"
Nov 03 20:30:27.551377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.549558    5659 cpu_manager.go:414] "containerMap: removing container" podUID="db088bce-259a-464c-953c-c2fe4d33582a" containerName="cleanup"
Nov 03 20:30:27.551377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.549565    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db088bce-259a-464c-953c-c2fe4d33582a" containerName="cleanup"
Nov 03 20:30:27.551377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.549584    5659 cpu_manager.go:414] "containerMap: removing container" podUID="659d88c4-b3c0-49d1-b799-7c746abdef13" containerName="cleanup"
Nov 03 20:30:27.551377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.549591    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659d88c4-b3c0-49d1-b799-7c746abdef13" containerName="cleanup"
Nov 03 20:30:27.551377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.549823    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="659d88c4-b3c0-49d1-b799-7c746abdef13" containerName="cleanup"
Nov 03 20:30:27.551377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.549842    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="db088bce-259a-464c-953c-c2fe4d33582a" containerName="cleanup"
Nov 03 20:30:27.551377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.549861    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="659d88c4-b3c0-49d1-b799-7c746abdef13" containerName="cleanup"
Nov 03 20:30:27.562013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.561986    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"
Nov 03 20:30:27.565477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.565071    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"]
Nov 03 20:30:27.627628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.627596    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fk64\" (UniqueName: \"kubernetes.io/projected/4729d882-2aaa-4224-ab95-aec46b29a2e7-kube-api-access-8fk64\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf\" (UID: \"4729d882-2aaa-4224-ab95-aec46b29a2e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"
Nov 03 20:30:27.627814 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.627741    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4729d882-2aaa-4224-ab95-aec46b29a2e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf\" (UID: \"4729d882-2aaa-4224-ab95-aec46b29a2e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"
Nov 03 20:30:27.628064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.628048    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn5pk\" (UniqueName: \"kubernetes.io/projected/659d88c4-b3c0-49d1-b799-7c746abdef13-kube-api-access-xn5pk\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:27.651676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.651635    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:27.743610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.743559    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fk64\" (UniqueName: \"kubernetes.io/projected/4729d882-2aaa-4224-ab95-aec46b29a2e7-kube-api-access-8fk64\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf\" (UID: \"4729d882-2aaa-4224-ab95-aec46b29a2e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"
Nov 03 20:30:27.743827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.743804    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4729d882-2aaa-4224-ab95-aec46b29a2e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf\" (UID: \"4729d882-2aaa-4224-ab95-aec46b29a2e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"
Nov 03 20:30:27.744518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.744461    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4729d882-2aaa-4224-ab95-aec46b29a2e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf\" (UID: \"4729d882-2aaa-4224-ab95-aec46b29a2e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"
Nov 03 20:30:27.756331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.756296    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fk64\" (UniqueName: \"kubernetes.io/projected/4729d882-2aaa-4224-ab95-aec46b29a2e7-kube-api-access-8fk64\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf\" (UID: \"4729d882-2aaa-4224-ab95-aec46b29a2e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"
Nov 03 20:30:27.880676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.880592    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"
Nov 03 20:30:27.978860 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:27.972931    5659 scope.go:122] "RemoveContainer" containerID="8fe02ef24f61bd94f9fa933bcdb837e914c88f67e27d5d35e5a39ed7f4d1f662"
Nov 03 20:30:28.179098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:28.179069    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"]
Nov 03 20:30:28.179636 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:28.179610    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4729d882_2aaa_4224_ab95_aec46b29a2e7.slice/crio-f7bf78ec2cfefa8a4916cee2a155f743f98921641f329527696e72753b2f4fbe WatchSource:0}: Error finding container f7bf78ec2cfefa8a4916cee2a155f743f98921641f329527696e72753b2f4fbe: Status 404 returned error can't find the container with id f7bf78ec2cfefa8a4916cee2a155f743f98921641f329527696e72753b2f4fbe
Nov 03 20:30:28.288635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:28.288600    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"
Nov 03 20:30:28.288812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:28.288630    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp" event={"ID":"659d88c4-b3c0-49d1-b799-7c746abdef13","Type":"ContainerDied","Data":"595e970565555b9f41209f01da3b0835b27add489f0bcbf31c79d0e22e8ff621"}
Nov 03 20:30:28.288812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:28.288689    5659 scope.go:122] "RemoveContainer" containerID="4757e4763bddcf6e6571e0950d5d84d38cbe9260bb810ff837475d0e0eb57b14"
Nov 03 20:30:28.290168 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:28.290134    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf" event={"ID":"4729d882-2aaa-4224-ab95-aec46b29a2e7","Type":"ContainerStarted","Data":"f7bf78ec2cfefa8a4916cee2a155f743f98921641f329527696e72753b2f4fbe"}
Nov 03 20:30:28.314117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:28.314085    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"]
Nov 03 20:30:28.322868 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:28.322833    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-74dfc6-29895630-x78vp"]
Nov 03 20:30:28.382214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:28.382114    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:29.194890 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:29.193336    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:29.302933 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:29.302902    5659 generic.go:356] "Generic (PLEG): container finished" podID="844f0645-fbd2-4496-9621-343de2fec107" containerID="89f1a5f54b73de7d3ed58cbdea7a0e75346342fc8455e994400ef8c2ab2a89b2" exitCode=60
Nov 03 20:30:29.303099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:29.302972    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9" event={"ID":"844f0645-fbd2-4496-9621-343de2fec107","Type":"ContainerDied","Data":"89f1a5f54b73de7d3ed58cbdea7a0e75346342fc8455e994400ef8c2ab2a89b2"}
Nov 03 20:30:29.303099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:29.303030    5659 scope.go:122] "RemoveContainer" containerID="8fe02ef24f61bd94f9fa933bcdb837e914c88f67e27d5d35e5a39ed7f4d1f662"
Nov 03 20:30:29.303450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:29.303431    5659 scope.go:122] "RemoveContainer" containerID="89f1a5f54b73de7d3ed58cbdea7a0e75346342fc8455e994400ef8c2ab2a89b2"
Nov 03 20:30:29.303822 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:29.303798    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9_odh-ai-gateway-infra(844f0645-fbd2-4496-9621-343de2fec107)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9" podUID="844f0645-fbd2-4496-9621-343de2fec107"
Nov 03 20:30:29.304853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:29.304834    5659 generic.go:356] "Generic (PLEG): container finished" podID="4729d882-2aaa-4224-ab95-aec46b29a2e7" containerID="ef79f1479cd2875a94a45fcedecb94e057b0fa5e991099b36d18c75dbe53eae6" exitCode=60
Nov 03 20:30:29.304941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:29.304891    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf" event={"ID":"4729d882-2aaa-4224-ab95-aec46b29a2e7","Type":"ContainerDied","Data":"ef79f1479cd2875a94a45fcedecb94e057b0fa5e991099b36d18c75dbe53eae6"}
Nov 03 20:30:29.305136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:29.305124    5659 scope.go:122] "RemoveContainer" containerID="ef79f1479cd2875a94a45fcedecb94e057b0fa5e991099b36d18c75dbe53eae6"
Nov 03 20:30:29.975474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:29.975439    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659d88c4-b3c0-49d1-b799-7c746abdef13" path="/var/lib/kubelet/pods/659d88c4-b3c0-49d1-b799-7c746abdef13/volumes"
Nov 03 20:30:30.021777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.021746    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:30.316818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.316735    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf" event={"ID":"4729d882-2aaa-4224-ab95-aec46b29a2e7","Type":"ContainerStarted","Data":"da0a38d89c676dd28364d9d4f42b58742a612780c262c997ce6ccacd9f8c4056"}
Nov 03 20:30:30.338726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.338697    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"]
Nov 03 20:30:30.340818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.340584    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf" podStartSLOduration=30.34056609 podStartE2EDuration="30.34056609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:30.334351391 +0000 UTC m=+3182.489253809" watchObservedRunningTime="2026-11-03 20:30:30.34056609 +0000 UTC m=+3182.495468510"
Nov 03 20:30:30.622345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.622278    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"
Nov 03 20:30:30.656025 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.655986    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"]
Nov 03 20:30:30.667963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.667927    5659 cpu_manager.go:414] "containerMap: removing container" podUID="844f0645-fbd2-4496-9621-343de2fec107" containerName="cleanup"
Nov 03 20:30:30.668169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.668150    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844f0645-fbd2-4496-9621-343de2fec107" containerName="cleanup"
Nov 03 20:30:30.668350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.668335    5659 cpu_manager.go:414] "containerMap: removing container" podUID="844f0645-fbd2-4496-9621-343de2fec107" containerName="cleanup"
Nov 03 20:30:30.668453 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.668441    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844f0645-fbd2-4496-9621-343de2fec107" containerName="cleanup"
Nov 03 20:30:30.668619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.668604    5659 cpu_manager.go:414] "containerMap: removing container" podUID="844f0645-fbd2-4496-9621-343de2fec107" containerName="cleanup"
Nov 03 20:30:30.668708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.668696    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844f0645-fbd2-4496-9621-343de2fec107" containerName="cleanup"
Nov 03 20:30:30.668811 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.668800    5659 cpu_manager.go:414] "containerMap: removing container" podUID="659d88c4-b3c0-49d1-b799-7c746abdef13" containerName="cleanup"
Nov 03 20:30:30.668891 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.668880    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659d88c4-b3c0-49d1-b799-7c746abdef13" containerName="cleanup"
Nov 03 20:30:30.669227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.669207    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="844f0645-fbd2-4496-9621-343de2fec107" containerName="cleanup"
Nov 03 20:30:30.669353 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.669340    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="659d88c4-b3c0-49d1-b799-7c746abdef13" containerName="cleanup"
Nov 03 20:30:30.669455 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.669444    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="844f0645-fbd2-4496-9621-343de2fec107" containerName="cleanup"
Nov 03 20:30:30.674815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.674794    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"]
Nov 03 20:30:30.674999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.674890    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"
Nov 03 20:30:30.720051 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.720021    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844f0645-fbd2-4496-9621-343de2fec107-kube-api-access-hwghk\" (UniqueName: \"kubernetes.io/projected/844f0645-fbd2-4496-9621-343de2fec107-kube-api-access-hwghk\") pod \"844f0645-fbd2-4496-9621-343de2fec107\" (UID: \"844f0645-fbd2-4496-9621-343de2fec107\") "
Nov 03 20:30:30.720195 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.720138    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844f0645-fbd2-4496-9621-343de2fec107-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844f0645-fbd2-4496-9621-343de2fec107-maas-api-service-ca\") pod \"844f0645-fbd2-4496-9621-343de2fec107\" (UID: \"844f0645-fbd2-4496-9621-343de2fec107\") "
Nov 03 20:30:30.720394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.720371    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356069a-b2fa-4a45-935f-9dc18deb9d54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc\" (UID: \"8356069a-b2fa-4a45-935f-9dc18deb9d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"
Nov 03 20:30:30.720642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.720614    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844f0645-fbd2-4496-9621-343de2fec107-maas-api-service-ca" pod "844f0645-fbd2-4496-9621-343de2fec107" (UID: "844f0645-fbd2-4496-9621-343de2fec107"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:30.720737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.720630    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p82jl\" (UniqueName: \"kubernetes.io/projected/8356069a-b2fa-4a45-935f-9dc18deb9d54-kube-api-access-p82jl\") pod \"maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc\" (UID: \"8356069a-b2fa-4a45-935f-9dc18deb9d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"
Nov 03 20:30:30.721055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.721034    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/844f0645-fbd2-4496-9621-343de2fec107-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:30.722421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.722399    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844f0645-fbd2-4496-9621-343de2fec107-kube-api-access-hwghk" pod "844f0645-fbd2-4496-9621-343de2fec107" (UID: "844f0645-fbd2-4496-9621-343de2fec107"). InnerVolumeSpecName "kube-api-access-hwghk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:30.749613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.749584    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:30.838513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.834283    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356069a-b2fa-4a45-935f-9dc18deb9d54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc\" (UID: \"8356069a-b2fa-4a45-935f-9dc18deb9d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"
Nov 03 20:30:30.838513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.834525    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82jl\" (UniqueName: \"kubernetes.io/projected/8356069a-b2fa-4a45-935f-9dc18deb9d54-kube-api-access-p82jl\") pod \"maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc\" (UID: \"8356069a-b2fa-4a45-935f-9dc18deb9d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"
Nov 03 20:30:30.838513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.834645    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwghk\" (UniqueName: \"kubernetes.io/projected/844f0645-fbd2-4496-9621-343de2fec107-kube-api-access-hwghk\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:30.838513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.835868    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356069a-b2fa-4a45-935f-9dc18deb9d54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc\" (UID: \"8356069a-b2fa-4a45-935f-9dc18deb9d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"
Nov 03 20:30:30.860844 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:30.860807    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p82jl\" (UniqueName: \"kubernetes.io/projected/8356069a-b2fa-4a45-935f-9dc18deb9d54-kube-api-access-p82jl\") pod \"maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc\" (UID: \"8356069a-b2fa-4a45-935f-9dc18deb9d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"
Nov 03 20:30:31.001065 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.001033    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"
Nov 03 20:30:31.315840 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:31.315811    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8356069a_b2fa_4a45_935f_9dc18deb9d54.slice/crio-f23817924866b233bc838ce4fff4cea22e61a51b2fce2bdb395ef7e881482c71 WatchSource:0}: Error finding container f23817924866b233bc838ce4fff4cea22e61a51b2fce2bdb395ef7e881482c71: Status 404 returned error can't find the container with id f23817924866b233bc838ce4fff4cea22e61a51b2fce2bdb395ef7e881482c71
Nov 03 20:30:31.316047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.316023    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"]
Nov 03 20:30:31.329350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.329325    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"
Nov 03 20:30:31.329465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.329375    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9" event={"ID":"844f0645-fbd2-4496-9621-343de2fec107","Type":"ContainerDied","Data":"9a50b1c3ee4ba045283c56f3d60581f3652c0b999a835d0a1db0228077a79bc9"}
Nov 03 20:30:31.329465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.329437    5659 scope.go:122] "RemoveContainer" containerID="89f1a5f54b73de7d3ed58cbdea7a0e75346342fc8455e994400ef8c2ab2a89b2"
Nov 03 20:30:31.331308 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.331277    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc" event={"ID":"8356069a-b2fa-4a45-935f-9dc18deb9d54","Type":"ContainerStarted","Data":"f23817924866b233bc838ce4fff4cea22e61a51b2fce2bdb395ef7e881482c71"}
Nov 03 20:30:31.333512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.333471    5659 generic.go:356] "Generic (PLEG): container finished" podID="4729d882-2aaa-4224-ab95-aec46b29a2e7" containerID="da0a38d89c676dd28364d9d4f42b58742a612780c262c997ce6ccacd9f8c4056" exitCode=60
Nov 03 20:30:31.333686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.333670    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf" event={"ID":"4729d882-2aaa-4224-ab95-aec46b29a2e7","Type":"ContainerDied","Data":"da0a38d89c676dd28364d9d4f42b58742a612780c262c997ce6ccacd9f8c4056"}
Nov 03 20:30:31.335707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.334103    5659 scope.go:122] "RemoveContainer" containerID="da0a38d89c676dd28364d9d4f42b58742a612780c262c997ce6ccacd9f8c4056"
Nov 03 20:30:31.335707 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:31.334793    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf_odh-ai-gateway-infra(4729d882-2aaa-4224-ab95-aec46b29a2e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf" podUID="4729d882-2aaa-4224-ab95-aec46b29a2e7"
Nov 03 20:30:31.345646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.345623    5659 scope.go:122] "RemoveContainer" containerID="ef79f1479cd2875a94a45fcedecb94e057b0fa5e991099b36d18c75dbe53eae6"
Nov 03 20:30:31.378722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.378680    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"]
Nov 03 20:30:31.386152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.386126    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bfc240-29895630-nrqr9"]
Nov 03 20:30:31.431511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.401401    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:31.542305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.542275    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:31.933906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.933807    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:31.978797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:31.978756    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844f0645-fbd2-4496-9621-343de2fec107" path="/var/lib/kubelet/pods/844f0645-fbd2-4496-9621-343de2fec107/volumes"
Nov 03 20:30:32.304609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:32.304576    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:32.347396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:32.347274    5659 scope.go:122] "RemoveContainer" containerID="da0a38d89c676dd28364d9d4f42b58742a612780c262c997ce6ccacd9f8c4056"
Nov 03 20:30:32.348058 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:32.348027    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf_odh-ai-gateway-infra(4729d882-2aaa-4224-ab95-aec46b29a2e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf" podUID="4729d882-2aaa-4224-ab95-aec46b29a2e7"
Nov 03 20:30:32.349194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:32.349161    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc" event={"ID":"8356069a-b2fa-4a45-935f-9dc18deb9d54","Type":"ContainerStarted","Data":"292cc302723a556b112bb94e1ef2d5283669dbe2af2d1bfbf4620dbcd73ecc58"}
Nov 03 20:30:32.349468 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:32.349452    5659 scope.go:122] "RemoveContainer" containerID="292cc302723a556b112bb94e1ef2d5283669dbe2af2d1bfbf4620dbcd73ecc58"
Nov 03 20:30:33.092168 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:33.092074    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:33.361546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:33.361438    5659 generic.go:356] "Generic (PLEG): container finished" podID="8356069a-b2fa-4a45-935f-9dc18deb9d54" containerID="292cc302723a556b112bb94e1ef2d5283669dbe2af2d1bfbf4620dbcd73ecc58" exitCode=60
Nov 03 20:30:33.361546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:33.361511    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc" event={"ID":"8356069a-b2fa-4a45-935f-9dc18deb9d54","Type":"ContainerDied","Data":"292cc302723a556b112bb94e1ef2d5283669dbe2af2d1bfbf4620dbcd73ecc58"}
Nov 03 20:30:33.362083 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:33.361550    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc" event={"ID":"8356069a-b2fa-4a45-935f-9dc18deb9d54","Type":"ContainerStarted","Data":"ab912947202d786def9cb410aa8b7a6defa4e8cd5d1fcf864fdf3e5ffa8de61f"}
Nov 03 20:30:33.383827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:33.383769    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc" podStartSLOduration=33.38375615 podStartE2EDuration="33.38375615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:33.380222283 +0000 UTC m=+3185.535124703" watchObservedRunningTime="2026-11-03 20:30:33.38375615 +0000 UTC m=+3185.538658568"
Nov 03 20:30:33.790504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:33.771212    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:34.376170 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:34.376135    5659 generic.go:356] "Generic (PLEG): container finished" podID="8356069a-b2fa-4a45-935f-9dc18deb9d54" containerID="ab912947202d786def9cb410aa8b7a6defa4e8cd5d1fcf864fdf3e5ffa8de61f" exitCode=60
Nov 03 20:30:34.376709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:34.376227    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc" event={"ID":"8356069a-b2fa-4a45-935f-9dc18deb9d54","Type":"ContainerDied","Data":"ab912947202d786def9cb410aa8b7a6defa4e8cd5d1fcf864fdf3e5ffa8de61f"}
Nov 03 20:30:34.376709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:34.376305    5659 scope.go:122] "RemoveContainer" containerID="292cc302723a556b112bb94e1ef2d5283669dbe2af2d1bfbf4620dbcd73ecc58"
Nov 03 20:30:34.376709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:34.376649    5659 scope.go:122] "RemoveContainer" containerID="ab912947202d786def9cb410aa8b7a6defa4e8cd5d1fcf864fdf3e5ffa8de61f"
Nov 03 20:30:34.377107 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:34.377085    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc_odh-ai-gateway-infra(8356069a-b2fa-4a45-935f-9dc18deb9d54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc" podUID="8356069a-b2fa-4a45-935f-9dc18deb9d54"
Nov 03 20:30:34.438337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:34.438298    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:35.164083 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:35.164046    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:35.387379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:35.387341    5659 scope.go:122] "RemoveContainer" containerID="ab912947202d786def9cb410aa8b7a6defa4e8cd5d1fcf864fdf3e5ffa8de61f"
Nov 03 20:30:35.387960 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:35.387927    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc_odh-ai-gateway-infra(8356069a-b2fa-4a45-935f-9dc18deb9d54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc" podUID="8356069a-b2fa-4a45-935f-9dc18deb9d54"
Nov 03 20:30:35.824682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:35.824648    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:35.961982 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:35.961942    5659 scope.go:122] "RemoveContainer" containerID="36d5655913b862378ff0c337314951b25c5f3977d696774bf3f68e148fc58df6"
Nov 03 20:30:35.962263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:35.962243    5659 scope.go:122] "RemoveContainer" containerID="710024563df86faadfe631844747b9db72429ec46d281f27ede7174186bc0d83"
Nov 03 20:30:36.500303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:36.500267    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:36.962203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:36.962164    5659 scope.go:122] "RemoveContainer" containerID="ef18ba699ab19799d1dbf82cd4f6942b414280110c39d9f4b9f3dfab048479f4"
Nov 03 20:30:37.285429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:37.285394    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:37.408102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:37.408063    5659 generic.go:356] "Generic (PLEG): container finished" podID="bde691e8-b602-449e-a763-a94adcb073c5" containerID="26e32613679ccb69cdf23a6f020955b89ddc6cd49628410b46c4601f31e89656" exitCode=60
Nov 03 20:30:37.408314 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:37.408136    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg" event={"ID":"bde691e8-b602-449e-a763-a94adcb073c5","Type":"ContainerDied","Data":"26e32613679ccb69cdf23a6f020955b89ddc6cd49628410b46c4601f31e89656"}
Nov 03 20:30:37.408314 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:37.408208    5659 scope.go:122] "RemoveContainer" containerID="36d5655913b862378ff0c337314951b25c5f3977d696774bf3f68e148fc58df6"
Nov 03 20:30:37.408734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:37.408714    5659 scope.go:122] "RemoveContainer" containerID="26e32613679ccb69cdf23a6f020955b89ddc6cd49628410b46c4601f31e89656"
Nov 03 20:30:37.409202 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:37.409142    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg_odh-ai-gateway-infra(bde691e8-b602-449e-a763-a94adcb073c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg" podUID="bde691e8-b602-449e-a763-a94adcb073c5"
Nov 03 20:30:37.411155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:37.411033    5659 generic.go:356] "Generic (PLEG): container finished" podID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c" containerID="c30fdaf7c2819150d24a6a42e045a152e01cd951baa095ad64360d6dc1f95529" exitCode=60
Nov 03 20:30:37.411155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:37.411120    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf" event={"ID":"24fc2329-33a9-4d19-a4ca-a1b0b83c893c","Type":"ContainerDied","Data":"c30fdaf7c2819150d24a6a42e045a152e01cd951baa095ad64360d6dc1f95529"}
Nov 03 20:30:37.411470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:37.411454    5659 scope.go:122] "RemoveContainer" containerID="c30fdaf7c2819150d24a6a42e045a152e01cd951baa095ad64360d6dc1f95529"
Nov 03 20:30:37.411754 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:37.411733    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf_odh-ai-gateway-infra(24fc2329-33a9-4d19-a4ca-a1b0b83c893c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf" podUID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c"
Nov 03 20:30:37.422383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:37.422362    5659 scope.go:122] "RemoveContainer" containerID="710024563df86faadfe631844747b9db72429ec46d281f27ede7174186bc0d83"
Nov 03 20:30:37.937838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:37.936583    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:38.020549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.020518    5659 scope.go:122] "RemoveContainer" containerID="8e81a8a8d43b9f09678b566342adb15ac116e3850732a7d4d2509ebd1c948dfa"
Nov 03 20:30:38.421689 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.421657    5659 generic.go:356] "Generic (PLEG): container finished" podID="8d2e7d4e-68f3-48a9-9fff-133223dadb65" containerID="7b16fb4f1e3abdf7fea3aac98067f4e42003d015244f898437149843b0c6d800" exitCode=22
Nov 03 20:30:38.421856 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.421725    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4" event={"ID":"8d2e7d4e-68f3-48a9-9fff-133223dadb65","Type":"ContainerDied","Data":"7b16fb4f1e3abdf7fea3aac98067f4e42003d015244f898437149843b0c6d800"}
Nov 03 20:30:38.421856 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.421788    5659 scope.go:122] "RemoveContainer" containerID="ef18ba699ab19799d1dbf82cd4f6942b414280110c39d9f4b9f3dfab048479f4"
Nov 03 20:30:38.422275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.422252    5659 scope.go:122] "RemoveContainer" containerID="7b16fb4f1e3abdf7fea3aac98067f4e42003d015244f898437149843b0c6d800"
Nov 03 20:30:38.422726 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:38.422690    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-x4nx4_odh-ai-gateway-infra(8d2e7d4e-68f3-48a9-9fff-133223dadb65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4" podUID="8d2e7d4e-68f3-48a9-9fff-133223dadb65"
Nov 03 20:30:38.454171 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.454138    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"]
Nov 03 20:30:38.464076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.464047    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"]
Nov 03 20:30:38.656396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.656356    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:38.788303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.781158    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"
Nov 03 20:30:38.845534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.845500    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"]
Nov 03 20:30:38.848757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.846698    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bde691e8-b602-449e-a763-a94adcb073c5" containerName="cleanup"
Nov 03 20:30:38.848757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.846721    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde691e8-b602-449e-a763-a94adcb073c5" containerName="cleanup"
Nov 03 20:30:38.848757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.846821    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bde691e8-b602-449e-a763-a94adcb073c5" containerName="cleanup"
Nov 03 20:30:38.848757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.846829    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde691e8-b602-449e-a763-a94adcb073c5" containerName="cleanup"
Nov 03 20:30:38.848757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.847013    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="844f0645-fbd2-4496-9621-343de2fec107" containerName="cleanup"
Nov 03 20:30:38.848757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.847038    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde691e8-b602-449e-a763-a94adcb073c5" containerName="cleanup"
Nov 03 20:30:38.848757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.847050    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde691e8-b602-449e-a763-a94adcb073c5" containerName="cleanup"
Nov 03 20:30:38.850191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.850027    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"
Nov 03 20:30:38.854339 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.854321    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"
Nov 03 20:30:38.869818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.869799    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"]
Nov 03 20:30:38.879433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.879360    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde691e8-b602-449e-a763-a94adcb073c5-kube-api-access-ggcrd\" (UniqueName: \"kubernetes.io/projected/bde691e8-b602-449e-a763-a94adcb073c5-kube-api-access-ggcrd\") pod \"bde691e8-b602-449e-a763-a94adcb073c5\" (UID: \"bde691e8-b602-449e-a763-a94adcb073c5\") "
Nov 03 20:30:38.879585 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.879467    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bde691e8-b602-449e-a763-a94adcb073c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde691e8-b602-449e-a763-a94adcb073c5-maas-api-service-ca\") pod \"bde691e8-b602-449e-a763-a94adcb073c5\" (UID: \"bde691e8-b602-449e-a763-a94adcb073c5\") "
Nov 03 20:30:38.880069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.880038    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde691e8-b602-449e-a763-a94adcb073c5-maas-api-service-ca" pod "bde691e8-b602-449e-a763-a94adcb073c5" (UID: "bde691e8-b602-449e-a763-a94adcb073c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:38.880300 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.880279    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde691e8-b602-449e-a763-a94adcb073c5-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:38.882662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.882637    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde691e8-b602-449e-a763-a94adcb073c5-kube-api-access-ggcrd" pod "bde691e8-b602-449e-a763-a94adcb073c5" (UID: "bde691e8-b602-449e-a763-a94adcb073c5"). InnerVolumeSpecName "kube-api-access-ggcrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:38.946952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.945609    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"]
Nov 03 20:30:38.946952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.946938    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bde691e8-b602-449e-a763-a94adcb073c5" containerName="cleanup"
Nov 03 20:30:38.947514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.946960    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde691e8-b602-449e-a763-a94adcb073c5" containerName="cleanup"
Nov 03 20:30:38.947514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.946978    5659 cpu_manager.go:414] "containerMap: removing container" podUID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c" containerName="cleanup"
Nov 03 20:30:38.947514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.946986    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c" containerName="cleanup"
Nov 03 20:30:38.947514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.946998    5659 cpu_manager.go:414] "containerMap: removing container" podUID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c" containerName="cleanup"
Nov 03 20:30:38.947514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.947006    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c" containerName="cleanup"
Nov 03 20:30:38.947514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.947310    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c" containerName="cleanup"
Nov 03 20:30:38.947514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.947329    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c" containerName="cleanup"
Nov 03 20:30:38.947514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.947355    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde691e8-b602-449e-a763-a94adcb073c5" containerName="cleanup"
Nov 03 20:30:38.952476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.951334    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"
Nov 03 20:30:38.956750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.956724    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"]
Nov 03 20:30:38.980819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.980793    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-kube-api-access-m5qnm\" (UniqueName: \"kubernetes.io/projected/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-kube-api-access-m5qnm\") pod \"24fc2329-33a9-4d19-a4ca-a1b0b83c893c\" (UID: \"24fc2329-33a9-4d19-a4ca-a1b0b83c893c\") "
Nov 03 20:30:38.981013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.980998    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-maas-api-service-ca\") pod \"24fc2329-33a9-4d19-a4ca-a1b0b83c893c\" (UID: \"24fc2329-33a9-4d19-a4ca-a1b0b83c893c\") "
Nov 03 20:30:38.981373 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.981344    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-maas-api-service-ca" pod "24fc2329-33a9-4d19-a4ca-a1b0b83c893c" (UID: "24fc2329-33a9-4d19-a4ca-a1b0b83c893c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:38.981472 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.981400    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-tfnfd\" (UID: \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"
Nov 03 20:30:38.981642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.981613    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-tfnfd\" (UID: \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"
Nov 03 20:30:38.981806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.981786    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmm8h\" (UniqueName: \"kubernetes.io/projected/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-kube-api-access-rmm8h\") pod \"maas-api-revoke-sub-f14db2bd-tfnfd\" (UID: \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"
Nov 03 20:30:38.982013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.981995    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggcrd\" (UniqueName: \"kubernetes.io/projected/bde691e8-b602-449e-a763-a94adcb073c5-kube-api-access-ggcrd\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:38.982077 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.982019    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:38.983314 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:38.983292    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-kube-api-access-m5qnm" pod "24fc2329-33a9-4d19-a4ca-a1b0b83c893c" (UID: "24fc2329-33a9-4d19-a4ca-a1b0b83c893c"). InnerVolumeSpecName "kube-api-access-m5qnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:39.083629 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.083587    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231e523c-84a7-4b9f-9f6c-6b9603566f86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb\" (UID: \"231e523c-84a7-4b9f-9f6c-6b9603566f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"
Nov 03 20:30:39.083789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.083655    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-tfnfd\" (UID: \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"
Nov 03 20:30:39.083854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.083827    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk8qh\" (UniqueName: \"kubernetes.io/projected/231e523c-84a7-4b9f-9f6c-6b9603566f86-kube-api-access-zk8qh\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb\" (UID: \"231e523c-84a7-4b9f-9f6c-6b9603566f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"
Nov 03 20:30:39.083944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.083927    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-tfnfd\" (UID: \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"
Nov 03 20:30:39.084205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.084183    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmm8h\" (UniqueName: \"kubernetes.io/projected/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-kube-api-access-rmm8h\") pod \"maas-api-revoke-sub-f14db2bd-tfnfd\" (UID: \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"
Nov 03 20:30:39.084357 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.084342    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qnm\" (UniqueName: \"kubernetes.io/projected/24fc2329-33a9-4d19-a4ca-a1b0b83c893c-kube-api-access-m5qnm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:39.084822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.084798    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-tfnfd\" (UID: \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"
Nov 03 20:30:39.086523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.086455    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-tfnfd\" (UID: \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"
Nov 03 20:30:39.096416 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.096389    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmm8h\" (UniqueName: \"kubernetes.io/projected/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-kube-api-access-rmm8h\") pod \"maas-api-revoke-sub-f14db2bd-tfnfd\" (UID: \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"
Nov 03 20:30:39.170344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.170310    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"
Nov 03 20:30:39.185598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.185570    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231e523c-84a7-4b9f-9f6c-6b9603566f86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb\" (UID: \"231e523c-84a7-4b9f-9f6c-6b9603566f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"
Nov 03 20:30:39.185708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.185679    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk8qh\" (UniqueName: \"kubernetes.io/projected/231e523c-84a7-4b9f-9f6c-6b9603566f86-kube-api-access-zk8qh\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb\" (UID: \"231e523c-84a7-4b9f-9f6c-6b9603566f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"
Nov 03 20:30:39.186405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.186359    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231e523c-84a7-4b9f-9f6c-6b9603566f86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb\" (UID: \"231e523c-84a7-4b9f-9f6c-6b9603566f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"
Nov 03 20:30:39.208439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.208403    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk8qh\" (UniqueName: \"kubernetes.io/projected/231e523c-84a7-4b9f-9f6c-6b9603566f86-kube-api-access-zk8qh\") pod \"maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb\" (UID: \"231e523c-84a7-4b9f-9f6c-6b9603566f86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"
Nov 03 20:30:39.277961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.277922    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"
Nov 03 20:30:39.440297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.440179    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg" event={"ID":"bde691e8-b602-449e-a763-a94adcb073c5","Type":"ContainerDied","Data":"d7f40a42c320a273f424a83d7fc4b50b8304c02ee2576b8d6279c216c64d15cb"}
Nov 03 20:30:39.440297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.440198    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"
Nov 03 20:30:39.441070 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.441046    5659 scope.go:122] "RemoveContainer" containerID="26e32613679ccb69cdf23a6f020955b89ddc6cd49628410b46c4601f31e89656"
Nov 03 20:30:39.443800 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.443775    5659 generic.go:356] "Generic (PLEG): container finished" podID="f4069450-679d-44cb-ab4e-44c7fea95e76" containerID="248b6e654784dc52d6b59511a18d8616f1742f91fac4dcbedb55dd34e296ade4" exitCode=22
Nov 03 20:30:39.443918 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.443826    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp" event={"ID":"f4069450-679d-44cb-ab4e-44c7fea95e76","Type":"ContainerDied","Data":"248b6e654784dc52d6b59511a18d8616f1742f91fac4dcbedb55dd34e296ade4"}
Nov 03 20:30:39.444390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.444363    5659 scope.go:122] "RemoveContainer" containerID="248b6e654784dc52d6b59511a18d8616f1742f91fac4dcbedb55dd34e296ade4"
Nov 03 20:30:39.445028 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:39.444998    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-q8zvp_odh-ai-gateway-infra(f4069450-679d-44cb-ab4e-44c7fea95e76)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp" podUID="f4069450-679d-44cb-ab4e-44c7fea95e76"
Nov 03 20:30:39.446605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.446580    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"
Nov 03 20:30:39.446988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.446578    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf" event={"ID":"24fc2329-33a9-4d19-a4ca-a1b0b83c893c","Type":"ContainerDied","Data":"ea52c1e1ad6f54c5d632dc5e1dc178a76e625ce76c30c7fdbe3a519185529233"}
Nov 03 20:30:39.465045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.464178    5659 scope.go:122] "RemoveContainer" containerID="8e81a8a8d43b9f09678b566342adb15ac116e3850732a7d4d2509ebd1c948dfa"
Nov 03 20:30:39.476664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.476477    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"]
Nov 03 20:30:39.483326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.483305    5659 scope.go:122] "RemoveContainer" containerID="c30fdaf7c2819150d24a6a42e045a152e01cd951baa095ad64360d6dc1f95529"
Nov 03 20:30:39.489260 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:39.489218    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e2428c_648d_4ef4_bed2_8b50adb40fe2.slice/crio-402ed13fbb07dab640a3390d640af84c12dd5910fa183a0b0ea40e86ff9e9db8 WatchSource:0}: Error finding container 402ed13fbb07dab640a3390d640af84c12dd5910fa183a0b0ea40e86ff9e9db8: Status 404 returned error can't find the container with id 402ed13fbb07dab640a3390d640af84c12dd5910fa183a0b0ea40e86ff9e9db8
Nov 03 20:30:39.494188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.494161    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"]
Nov 03 20:30:39.532384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.532353    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"]
Nov 03 20:30:39.540434 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.540400    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-836265-29895630-fcctf"]
Nov 03 20:30:39.547698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.547674    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"]
Nov 03 20:30:39.556370 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.556341    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895630-4n4cg"]
Nov 03 20:30:39.566553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.566351    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:39.692507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.692411    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"]
Nov 03 20:30:39.751730 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.751706    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"
Nov 03 20:30:39.788857 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.788819    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"]
Nov 03 20:30:39.790421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.790391    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8d2e7d4e-68f3-48a9-9fff-133223dadb65" containerName="revoke-keys"
Nov 03 20:30:39.790620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.790600    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2e7d4e-68f3-48a9-9fff-133223dadb65" containerName="revoke-keys"
Nov 03 20:30:39.790767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.790754    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8d2e7d4e-68f3-48a9-9fff-133223dadb65" containerName="revoke-keys"
Nov 03 20:30:39.790854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.790842    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2e7d4e-68f3-48a9-9fff-133223dadb65" containerName="revoke-keys"
Nov 03 20:30:39.790985 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.790973    5659 cpu_manager.go:414] "containerMap: removing container" podUID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c" containerName="cleanup"
Nov 03 20:30:39.791068 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.791057    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c" containerName="cleanup"
Nov 03 20:30:39.791360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.791342    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2e7d4e-68f3-48a9-9fff-133223dadb65" containerName="revoke-keys"
Nov 03 20:30:39.791505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.791475    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2e7d4e-68f3-48a9-9fff-133223dadb65" containerName="revoke-keys"
Nov 03 20:30:39.791624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.791613    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c" containerName="cleanup"
Nov 03 20:30:39.795004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.794980    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2e7d4e-68f3-48a9-9fff-133223dadb65-kube-api-access-6rzmd\" (UniqueName: \"kubernetes.io/projected/8d2e7d4e-68f3-48a9-9fff-133223dadb65-kube-api-access-6rzmd\") pod \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\" (UID: \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\") "
Nov 03 20:30:39.795364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.795344    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-openshift-ca\") pod \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\" (UID: \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\") "
Nov 03 20:30:39.795624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.795603    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-xks-ca\") pod \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\" (UID: \"8d2e7d4e-68f3-48a9-9fff-133223dadb65\") "
Nov 03 20:30:39.797169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.797145    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-openshift-ca" pod "8d2e7d4e-68f3-48a9-9fff-133223dadb65" (UID: "8d2e7d4e-68f3-48a9-9fff-133223dadb65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:39.798190 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.798172    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"]
Nov 03 20:30:39.798396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.798378    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"
Nov 03 20:30:39.801947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.801921    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-xks-ca" pod "8d2e7d4e-68f3-48a9-9fff-133223dadb65" (UID: "8d2e7d4e-68f3-48a9-9fff-133223dadb65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:39.813975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.813944    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2e7d4e-68f3-48a9-9fff-133223dadb65-kube-api-access-6rzmd" pod "8d2e7d4e-68f3-48a9-9fff-133223dadb65" (UID: "8d2e7d4e-68f3-48a9-9fff-133223dadb65"). InnerVolumeSpecName "kube-api-access-6rzmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:39.897904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.897873    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s22s\" (UniqueName: \"kubernetes.io/projected/a15e208a-f352-452c-b139-9526f846d5ab-kube-api-access-8s22s\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z\" (UID: \"a15e208a-f352-452c-b139-9526f846d5ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"
Nov 03 20:30:39.898074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.898014    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15e208a-f352-452c-b139-9526f846d5ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z\" (UID: \"a15e208a-f352-452c-b139-9526f846d5ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"
Nov 03 20:30:39.898117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.898080    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:39.898157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.898119    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rzmd\" (UniqueName: \"kubernetes.io/projected/8d2e7d4e-68f3-48a9-9fff-133223dadb65-kube-api-access-6rzmd\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:39.898157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.898141    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2e7d4e-68f3-48a9-9fff-133223dadb65-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:39.974464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.974428    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24fc2329-33a9-4d19-a4ca-a1b0b83c893c" path="/var/lib/kubelet/pods/24fc2329-33a9-4d19-a4ca-a1b0b83c893c/volumes"
Nov 03 20:30:39.975177 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.975158    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde691e8-b602-449e-a763-a94adcb073c5" path="/var/lib/kubelet/pods/bde691e8-b602-449e-a763-a94adcb073c5/volumes"
Nov 03 20:30:39.999264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.999222    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s22s\" (UniqueName: \"kubernetes.io/projected/a15e208a-f352-452c-b139-9526f846d5ab-kube-api-access-8s22s\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z\" (UID: \"a15e208a-f352-452c-b139-9526f846d5ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"
Nov 03 20:30:39.999430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:39.999410    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15e208a-f352-452c-b139-9526f846d5ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z\" (UID: \"a15e208a-f352-452c-b139-9526f846d5ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"
Nov 03 20:30:40.000106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.000082    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15e208a-f352-452c-b139-9526f846d5ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z\" (UID: \"a15e208a-f352-452c-b139-9526f846d5ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"
Nov 03 20:30:40.032431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.032404    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s22s\" (UniqueName: \"kubernetes.io/projected/a15e208a-f352-452c-b139-9526f846d5ab-kube-api-access-8s22s\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z\" (UID: \"a15e208a-f352-452c-b139-9526f846d5ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"
Nov 03 20:30:40.121900 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.121865    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"
Nov 03 20:30:40.232798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.232596    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:40.397327 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.397297    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:40.422432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.422397    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"]
Nov 03 20:30:40.422704 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:40.422666    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15e208a_f352_452c_b139_9526f846d5ab.slice/crio-5030333b675c5fa6bb317804a3f5cec8626f20487ec27bb1df80caa660e19866 WatchSource:0}: Error finding container 5030333b675c5fa6bb317804a3f5cec8626f20487ec27bb1df80caa660e19866: Status 404 returned error can't find the container with id 5030333b675c5fa6bb317804a3f5cec8626f20487ec27bb1df80caa660e19866
Nov 03 20:30:40.461947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.461902    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb" event={"ID":"231e523c-84a7-4b9f-9f6c-6b9603566f86","Type":"ContainerStarted","Data":"df7013ce8e28acd2f6daa9842f34b34261cd498aa1031d2e6d884b2b8d8b1e22"}
Nov 03 20:30:40.462090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.461965    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb" event={"ID":"231e523c-84a7-4b9f-9f6c-6b9603566f86","Type":"ContainerStarted","Data":"90fcfd1e019b1e00373d3ec53d04a370cb72b2af9d98323c429efdc764e022b9"}
Nov 03 20:30:40.466606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.466579    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd" event={"ID":"d6e2428c-648d-4ef4-bed2-8b50adb40fe2","Type":"ContainerStarted","Data":"c4c9ec664c35d6050fd8b50988f48fe275de43c50da65384a1dbeecf0ef570a2"}
Nov 03 20:30:40.466728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.466628    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd" event={"ID":"d6e2428c-648d-4ef4-bed2-8b50adb40fe2","Type":"ContainerStarted","Data":"402ed13fbb07dab640a3390d640af84c12dd5910fa183a0b0ea40e86ff9e9db8"}
Nov 03 20:30:40.468173 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.468150    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z" event={"ID":"a15e208a-f352-452c-b139-9526f846d5ab","Type":"ContainerStarted","Data":"5030333b675c5fa6bb317804a3f5cec8626f20487ec27bb1df80caa660e19866"}
Nov 03 20:30:40.469531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.469507    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4" event={"ID":"8d2e7d4e-68f3-48a9-9fff-133223dadb65","Type":"ContainerDied","Data":"75e9ebb9847757987f0508032a07734efa3554d8543c7fb727b69e53f3f40ca6"}
Nov 03 20:30:40.469610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.469557    5659 scope.go:122] "RemoveContainer" containerID="7b16fb4f1e3abdf7fea3aac98067f4e42003d015244f898437149843b0c6d800"
Nov 03 20:30:40.469610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.469516    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"
Nov 03 20:30:40.486313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.486271    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb" podStartSLOduration=40.486257677 podStartE2EDuration="40.486257677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:40.48068927 +0000 UTC m=+3192.635591689" watchObservedRunningTime="2026-11-03 20:30:40.486257677 +0000 UTC m=+3192.641160093"
Nov 03 20:30:40.494950 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.494925    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"]
Nov 03 20:30:40.506970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.506927    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd" podStartSLOduration=8.506914144 podStartE2EDuration="8.506914144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:40.501321084 +0000 UTC m=+3192.656223503" watchObservedRunningTime="2026-11-03 20:30:40.506914144 +0000 UTC m=+3192.661816557"
Nov 03 20:30:40.538429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.538404    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"]
Nov 03 20:30:40.546121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.546093    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-x4nx4"]
Nov 03 20:30:40.696916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.696883    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"
Nov 03 20:30:40.723577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.723524    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4069450-679d-44cb-ab4e-44c7fea95e76-kube-api-access-jfpq2\" (UniqueName: \"kubernetes.io/projected/f4069450-679d-44cb-ab4e-44c7fea95e76-kube-api-access-jfpq2\") pod \"f4069450-679d-44cb-ab4e-44c7fea95e76\" (UID: \"f4069450-679d-44cb-ab4e-44c7fea95e76\") "
Nov 03 20:30:40.723731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.723667    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-openshift-ca\") pod \"f4069450-679d-44cb-ab4e-44c7fea95e76\" (UID: \"f4069450-679d-44cb-ab4e-44c7fea95e76\") "
Nov 03 20:30:40.723970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.723951    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-xks-ca\") pod \"f4069450-679d-44cb-ab4e-44c7fea95e76\" (UID: \"f4069450-679d-44cb-ab4e-44c7fea95e76\") "
Nov 03 20:30:40.724161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.724132    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-openshift-ca" pod "f4069450-679d-44cb-ab4e-44c7fea95e76" (UID: "f4069450-679d-44cb-ab4e-44c7fea95e76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:40.724781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.724755    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:40.727192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.727165    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-xks-ca" pod "f4069450-679d-44cb-ab4e-44c7fea95e76" (UID: "f4069450-679d-44cb-ab4e-44c7fea95e76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:40.727577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.727556    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4069450-679d-44cb-ab4e-44c7fea95e76-kube-api-access-jfpq2" pod "f4069450-679d-44cb-ab4e-44c7fea95e76" (UID: "f4069450-679d-44cb-ab4e-44c7fea95e76"). InnerVolumeSpecName "kube-api-access-jfpq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:40.743321 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.743113    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"]
Nov 03 20:30:40.744195 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.744165    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f4069450-679d-44cb-ab4e-44c7fea95e76" containerName="revoke-keys"
Nov 03 20:30:40.744195 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.744186    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4069450-679d-44cb-ab4e-44c7fea95e76" containerName="revoke-keys"
Nov 03 20:30:40.744195 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.744195    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f4069450-679d-44cb-ab4e-44c7fea95e76" containerName="revoke-keys"
Nov 03 20:30:40.744195 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.744200    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4069450-679d-44cb-ab4e-44c7fea95e76" containerName="revoke-keys"
Nov 03 20:30:40.744447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.744212    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8d2e7d4e-68f3-48a9-9fff-133223dadb65" containerName="revoke-keys"
Nov 03 20:30:40.744447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.744217    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2e7d4e-68f3-48a9-9fff-133223dadb65" containerName="revoke-keys"
Nov 03 20:30:40.744447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.744239    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f4069450-679d-44cb-ab4e-44c7fea95e76" containerName="revoke-keys"
Nov 03 20:30:40.744447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.744244    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4069450-679d-44cb-ab4e-44c7fea95e76" containerName="revoke-keys"
Nov 03 20:30:40.744447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.744385    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4069450-679d-44cb-ab4e-44c7fea95e76" containerName="revoke-keys"
Nov 03 20:30:40.744447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.744391    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4069450-679d-44cb-ab4e-44c7fea95e76" containerName="revoke-keys"
Nov 03 20:30:40.744447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.744402    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2e7d4e-68f3-48a9-9fff-133223dadb65" containerName="revoke-keys"
Nov 03 20:30:40.747660 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.747636    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"
Nov 03 20:30:40.759678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.759655    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"]
Nov 03 20:30:40.825751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.825723    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88035b6-4c8a-4c03-b803-e897a2c86818-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l\" (UID: \"b88035b6-4c8a-4c03-b803-e897a2c86818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"
Nov 03 20:30:40.825956 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.825933    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmxnc\" (UniqueName: \"kubernetes.io/projected/b88035b6-4c8a-4c03-b803-e897a2c86818-kube-api-access-rmxnc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l\" (UID: \"b88035b6-4c8a-4c03-b803-e897a2c86818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"
Nov 03 20:30:40.826119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.826106    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4069450-679d-44cb-ab4e-44c7fea95e76-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:40.826158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.826122    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfpq2\" (UniqueName: \"kubernetes.io/projected/f4069450-679d-44cb-ab4e-44c7fea95e76-kube-api-access-jfpq2\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:40.928155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.928114    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88035b6-4c8a-4c03-b803-e897a2c86818-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l\" (UID: \"b88035b6-4c8a-4c03-b803-e897a2c86818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"
Nov 03 20:30:40.928631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.928386    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmxnc\" (UniqueName: \"kubernetes.io/projected/b88035b6-4c8a-4c03-b803-e897a2c86818-kube-api-access-rmxnc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l\" (UID: \"b88035b6-4c8a-4c03-b803-e897a2c86818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"
Nov 03 20:30:40.928738 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.928701    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88035b6-4c8a-4c03-b803-e897a2c86818-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l\" (UID: \"b88035b6-4c8a-4c03-b803-e897a2c86818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"
Nov 03 20:30:40.942828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:40.942802    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmxnc\" (UniqueName: \"kubernetes.io/projected/b88035b6-4c8a-4c03-b803-e897a2c86818-kube-api-access-rmxnc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l\" (UID: \"b88035b6-4c8a-4c03-b803-e897a2c86818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"
Nov 03 20:30:41.001701 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.001626    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:41.122087 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.122052    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"
Nov 03 20:30:41.415223 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.415191    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"]
Nov 03 20:30:41.415775 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:41.415746    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88035b6_4c8a_4c03_b803_e897a2c86818.slice/crio-e470143f18f63dc61a78b15c4943b7051ffadddfd4b6f849b4d63b40fb6af387 WatchSource:0}: Error finding container e470143f18f63dc61a78b15c4943b7051ffadddfd4b6f849b4d63b40fb6af387: Status 404 returned error can't find the container with id e470143f18f63dc61a78b15c4943b7051ffadddfd4b6f849b4d63b40fb6af387
Nov 03 20:30:41.478919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.478881    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z" event={"ID":"a15e208a-f352-452c-b139-9526f846d5ab","Type":"ContainerStarted","Data":"4359b900c903ca5af567d2fa99591f0990bc3aaedae35a49b174502fa8b7910f"}
Nov 03 20:30:41.481554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.481522    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l" event={"ID":"b88035b6-4c8a-4c03-b803-e897a2c86818","Type":"ContainerStarted","Data":"e470143f18f63dc61a78b15c4943b7051ffadddfd4b6f849b4d63b40fb6af387"}
Nov 03 20:30:41.483454 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.483427    5659 generic.go:356] "Generic (PLEG): container finished" podID="231e523c-84a7-4b9f-9f6c-6b9603566f86" containerID="df7013ce8e28acd2f6daa9842f34b34261cd498aa1031d2e6d884b2b8d8b1e22" exitCode=60
Nov 03 20:30:41.483605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.483566    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb" event={"ID":"231e523c-84a7-4b9f-9f6c-6b9603566f86","Type":"ContainerDied","Data":"df7013ce8e28acd2f6daa9842f34b34261cd498aa1031d2e6d884b2b8d8b1e22"}
Nov 03 20:30:41.484135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.484109    5659 scope.go:122] "RemoveContainer" containerID="df7013ce8e28acd2f6daa9842f34b34261cd498aa1031d2e6d884b2b8d8b1e22"
Nov 03 20:30:41.485980 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.485962    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"
Nov 03 20:30:41.486081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.485990    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp" event={"ID":"f4069450-679d-44cb-ab4e-44c7fea95e76","Type":"ContainerDied","Data":"9bbe1ce0057512fdb828cfc4ecfded07448a1092225c2dacff418cb01b404289"}
Nov 03 20:30:41.486081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.486047    5659 scope.go:122] "RemoveContainer" containerID="248b6e654784dc52d6b59511a18d8616f1742f91fac4dcbedb55dd34e296ade4"
Nov 03 20:30:41.495744 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.495718    5659 generic.go:356] "Generic (PLEG): container finished" podID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2" containerID="c4c9ec664c35d6050fd8b50988f48fe275de43c50da65384a1dbeecf0ef570a2" exitCode=22
Nov 03 20:30:41.495842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.495764    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd" event={"ID":"d6e2428c-648d-4ef4-bed2-8b50adb40fe2","Type":"ContainerDied","Data":"c4c9ec664c35d6050fd8b50988f48fe275de43c50da65384a1dbeecf0ef570a2"}
Nov 03 20:30:41.496129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.496110    5659 scope.go:122] "RemoveContainer" containerID="c4c9ec664c35d6050fd8b50988f48fe275de43c50da65384a1dbeecf0ef570a2"
Nov 03 20:30:41.508896 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.508853    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z" podStartSLOduration=41.508840229 podStartE2EDuration="41.508840229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:41.502558887 +0000 UTC m=+3193.657461304" watchObservedRunningTime="2026-11-03 20:30:41.508840229 +0000 UTC m=+3193.663742673"
Nov 03 20:30:41.543239 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.543185    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"]
Nov 03 20:30:41.549430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.549399    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-q8zvp"]
Nov 03 20:30:41.672585 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.672324    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:41.972052 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.972023    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2e7d4e-68f3-48a9-9fff-133223dadb65" path="/var/lib/kubelet/pods/8d2e7d4e-68f3-48a9-9fff-133223dadb65/volumes"
Nov 03 20:30:41.972578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:41.972561    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4069450-679d-44cb-ab4e-44c7fea95e76" path="/var/lib/kubelet/pods/f4069450-679d-44cb-ab4e-44c7fea95e76/volumes"
Nov 03 20:30:42.340552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:42.340522    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:42.525533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:42.515405    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l" event={"ID":"b88035b6-4c8a-4c03-b803-e897a2c86818","Type":"ContainerStarted","Data":"0459f0269ac90da2a0a5c123943fe563c0968caac694091659fe835f3b268aea"}
Nov 03 20:30:42.525533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:42.518386    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb" event={"ID":"231e523c-84a7-4b9f-9f6c-6b9603566f86","Type":"ContainerStarted","Data":"884d016e21ff67955fb3b9ea6fe77f1d56ab897fdf56a36fde75ea3fd72a60ff"}
Nov 03 20:30:42.525533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:42.524876    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd" event={"ID":"d6e2428c-648d-4ef4-bed2-8b50adb40fe2","Type":"ContainerStarted","Data":"ef4a0fdfc002af3edda96ca26de19c5cc2e29d2e3acc68f1b0ae803fa533223d"}
Nov 03 20:30:42.527432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:42.527402    5659 generic.go:356] "Generic (PLEG): container finished" podID="a15e208a-f352-452c-b139-9526f846d5ab" containerID="4359b900c903ca5af567d2fa99591f0990bc3aaedae35a49b174502fa8b7910f" exitCode=60
Nov 03 20:30:42.527632 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:42.527530    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z" event={"ID":"a15e208a-f352-452c-b139-9526f846d5ab","Type":"ContainerDied","Data":"4359b900c903ca5af567d2fa99591f0990bc3aaedae35a49b174502fa8b7910f"}
Nov 03 20:30:42.527882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:42.527864    5659 scope.go:122] "RemoveContainer" containerID="4359b900c903ca5af567d2fa99591f0990bc3aaedae35a49b174502fa8b7910f"
Nov 03 20:30:42.568377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:42.568314    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l" podStartSLOduration=42.568296438 podStartE2EDuration="42.568296438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:42.556161992 +0000 UTC m=+3194.711064426" watchObservedRunningTime="2026-11-03 20:30:42.568296438 +0000 UTC m=+3194.723198856"
Nov 03 20:30:42.594099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:42.593919    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:42.962198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:42.962157    5659 scope.go:122] "RemoveContainer" containerID="da0a38d89c676dd28364d9d4f42b58742a612780c262c997ce6ccacd9f8c4056"
Nov 03 20:30:43.232685 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.232638    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:43.542576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.542537    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf" event={"ID":"4729d882-2aaa-4224-ab95-aec46b29a2e7","Type":"ContainerStarted","Data":"a05c7eb63aef8bac118f05060f6471a4b399049a308c334b63f572d2feb49d46"}
Nov 03 20:30:43.544828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.544792    5659 generic.go:356] "Generic (PLEG): container finished" podID="b88035b6-4c8a-4c03-b803-e897a2c86818" containerID="0459f0269ac90da2a0a5c123943fe563c0968caac694091659fe835f3b268aea" exitCode=60
Nov 03 20:30:43.544966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.544870    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l" event={"ID":"b88035b6-4c8a-4c03-b803-e897a2c86818","Type":"ContainerDied","Data":"0459f0269ac90da2a0a5c123943fe563c0968caac694091659fe835f3b268aea"}
Nov 03 20:30:43.545202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.545187    5659 scope.go:122] "RemoveContainer" containerID="0459f0269ac90da2a0a5c123943fe563c0968caac694091659fe835f3b268aea"
Nov 03 20:30:43.547348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.547325    5659 generic.go:356] "Generic (PLEG): container finished" podID="231e523c-84a7-4b9f-9f6c-6b9603566f86" containerID="884d016e21ff67955fb3b9ea6fe77f1d56ab897fdf56a36fde75ea3fd72a60ff" exitCode=60
Nov 03 20:30:43.547458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.547369    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb" event={"ID":"231e523c-84a7-4b9f-9f6c-6b9603566f86","Type":"ContainerDied","Data":"884d016e21ff67955fb3b9ea6fe77f1d56ab897fdf56a36fde75ea3fd72a60ff"}
Nov 03 20:30:43.547458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.547439    5659 scope.go:122] "RemoveContainer" containerID="df7013ce8e28acd2f6daa9842f34b34261cd498aa1031d2e6d884b2b8d8b1e22"
Nov 03 20:30:43.548015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.547996    5659 scope.go:122] "RemoveContainer" containerID="884d016e21ff67955fb3b9ea6fe77f1d56ab897fdf56a36fde75ea3fd72a60ff"
Nov 03 20:30:43.549043 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:43.548394    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb_odh-ai-gateway-infra(231e523c-84a7-4b9f-9f6c-6b9603566f86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb" podUID="231e523c-84a7-4b9f-9f6c-6b9603566f86"
Nov 03 20:30:43.551375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.551351    5659 generic.go:356] "Generic (PLEG): container finished" podID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2" containerID="ef4a0fdfc002af3edda96ca26de19c5cc2e29d2e3acc68f1b0ae803fa533223d" exitCode=22
Nov 03 20:30:43.551507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.551437    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd" event={"ID":"d6e2428c-648d-4ef4-bed2-8b50adb40fe2","Type":"ContainerDied","Data":"ef4a0fdfc002af3edda96ca26de19c5cc2e29d2e3acc68f1b0ae803fa533223d"}
Nov 03 20:30:43.551800 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.551780    5659 scope.go:122] "RemoveContainer" containerID="ef4a0fdfc002af3edda96ca26de19c5cc2e29d2e3acc68f1b0ae803fa533223d"
Nov 03 20:30:43.552604 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:43.552207    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-tfnfd_odh-ai-gateway-infra(d6e2428c-648d-4ef4-bed2-8b50adb40fe2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2"
Nov 03 20:30:43.554326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.554304    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z" event={"ID":"a15e208a-f352-452c-b139-9526f846d5ab","Type":"ContainerStarted","Data":"10552455d97167d331ad974dfc9f03db46c2f580a755803f5fde776f9550e6b5"}
Nov 03 20:30:43.580740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.573896    5659 scope.go:122] "RemoveContainer" containerID="c4c9ec664c35d6050fd8b50988f48fe275de43c50da65384a1dbeecf0ef570a2"
Nov 03 20:30:43.963472 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:43.962784    5659 scope.go:122] "RemoveContainer" containerID="ab912947202d786def9cb410aa8b7a6defa4e8cd5d1fcf864fdf3e5ffa8de61f"
Nov 03 20:30:44.082120 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.080948    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:44.566144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.566114    5659 generic.go:356] "Generic (PLEG): container finished" podID="4729d882-2aaa-4224-ab95-aec46b29a2e7" containerID="a05c7eb63aef8bac118f05060f6471a4b399049a308c334b63f572d2feb49d46" exitCode=60
Nov 03 20:30:44.566620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.566187    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf" event={"ID":"4729d882-2aaa-4224-ab95-aec46b29a2e7","Type":"ContainerDied","Data":"a05c7eb63aef8bac118f05060f6471a4b399049a308c334b63f572d2feb49d46"}
Nov 03 20:30:44.566620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.566259    5659 scope.go:122] "RemoveContainer" containerID="da0a38d89c676dd28364d9d4f42b58742a612780c262c997ce6ccacd9f8c4056"
Nov 03 20:30:44.567102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.567082    5659 scope.go:122] "RemoveContainer" containerID="a05c7eb63aef8bac118f05060f6471a4b399049a308c334b63f572d2feb49d46"
Nov 03 20:30:44.569226 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:44.567642    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf_odh-ai-gateway-infra(4729d882-2aaa-4224-ab95-aec46b29a2e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf" podUID="4729d882-2aaa-4224-ab95-aec46b29a2e7"
Nov 03 20:30:44.569226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.569167    5659 generic.go:356] "Generic (PLEG): container finished" podID="b88035b6-4c8a-4c03-b803-e897a2c86818" containerID="ff7d07e674fd3e622c2d85ebb06069df5aae590dec26df3439672ea210b2b59f" exitCode=60
Nov 03 20:30:44.569422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.569242    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l" event={"ID":"b88035b6-4c8a-4c03-b803-e897a2c86818","Type":"ContainerDied","Data":"ff7d07e674fd3e622c2d85ebb06069df5aae590dec26df3439672ea210b2b59f"}
Nov 03 20:30:44.569682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.569662    5659 scope.go:122] "RemoveContainer" containerID="ff7d07e674fd3e622c2d85ebb06069df5aae590dec26df3439672ea210b2b59f"
Nov 03 20:30:44.570164 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:44.570139    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l_odh-ai-gateway-infra(b88035b6-4c8a-4c03-b803-e897a2c86818)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l" podUID="b88035b6-4c8a-4c03-b803-e897a2c86818"
Nov 03 20:30:44.571858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.571823    5659 scope.go:122] "RemoveContainer" containerID="884d016e21ff67955fb3b9ea6fe77f1d56ab897fdf56a36fde75ea3fd72a60ff"
Nov 03 20:30:44.572322 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:44.572282    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb_odh-ai-gateway-infra(231e523c-84a7-4b9f-9f6c-6b9603566f86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb" podUID="231e523c-84a7-4b9f-9f6c-6b9603566f86"
Nov 03 20:30:44.575199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.575180    5659 scope.go:122] "RemoveContainer" containerID="ef4a0fdfc002af3edda96ca26de19c5cc2e29d2e3acc68f1b0ae803fa533223d"
Nov 03 20:30:44.575592 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:44.575563    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-tfnfd_odh-ai-gateway-infra(d6e2428c-648d-4ef4-bed2-8b50adb40fe2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2"
Nov 03 20:30:44.576906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.576884    5659 generic.go:356] "Generic (PLEG): container finished" podID="a15e208a-f352-452c-b139-9526f846d5ab" containerID="10552455d97167d331ad974dfc9f03db46c2f580a755803f5fde776f9550e6b5" exitCode=60
Nov 03 20:30:44.576990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.576967    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z" event={"ID":"a15e208a-f352-452c-b139-9526f846d5ab","Type":"ContainerDied","Data":"10552455d97167d331ad974dfc9f03db46c2f580a755803f5fde776f9550e6b5"}
Nov 03 20:30:44.577249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.577229    5659 scope.go:122] "RemoveContainer" containerID="10552455d97167d331ad974dfc9f03db46c2f580a755803f5fde776f9550e6b5"
Nov 03 20:30:44.577640 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:44.577612    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z_odh-ai-gateway-infra(a15e208a-f352-452c-b139-9526f846d5ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z" podUID="a15e208a-f352-452c-b139-9526f846d5ab"
Nov 03 20:30:44.579021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.579001    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc" event={"ID":"8356069a-b2fa-4a45-935f-9dc18deb9d54","Type":"ContainerStarted","Data":"776d40814d51646d75d9d81f15ce79de12db78d7a867dd6b68205ded835ce2a7"}
Nov 03 20:30:44.582014 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.581994    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"]
Nov 03 20:30:44.596533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.596509    5659 scope.go:122] "RemoveContainer" containerID="0459f0269ac90da2a0a5c123943fe563c0968caac694091659fe835f3b268aea"
Nov 03 20:30:44.609418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.609393    5659 scope.go:122] "RemoveContainer" containerID="4359b900c903ca5af567d2fa99591f0990bc3aaedae35a49b174502fa8b7910f"
Nov 03 20:30:44.930415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:44.930340    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:45.219255 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:45.219215    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:30:45.519128 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.519049    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"]
Nov 03 20:30:45.519344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.519237    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c" podUID="f421d2e3-252b-45a6-b625-838fa53361c2" containerName="istio-proxy" containerID="cri-o://2b95e4f4a0f160d5565ba937a78dafb461bb5961ee8fdd8a0769311d6ec8df93" gracePeriod=30
Nov 03 20:30:45.595226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.595193    5659 scope.go:122] "RemoveContainer" containerID="10552455d97167d331ad974dfc9f03db46c2f580a755803f5fde776f9550e6b5"
Nov 03 20:30:45.598589 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:45.595748    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z_odh-ai-gateway-infra(a15e208a-f352-452c-b139-9526f846d5ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z" podUID="a15e208a-f352-452c-b139-9526f846d5ab"
Nov 03 20:30:45.598589 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.597478    5659 generic.go:356] "Generic (PLEG): container finished" podID="8356069a-b2fa-4a45-935f-9dc18deb9d54" containerID="776d40814d51646d75d9d81f15ce79de12db78d7a867dd6b68205ded835ce2a7" exitCode=60
Nov 03 20:30:45.598589 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.597657    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc" event={"ID":"8356069a-b2fa-4a45-935f-9dc18deb9d54","Type":"ContainerDied","Data":"776d40814d51646d75d9d81f15ce79de12db78d7a867dd6b68205ded835ce2a7"}
Nov 03 20:30:45.598589 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.597708    5659 scope.go:122] "RemoveContainer" containerID="ab912947202d786def9cb410aa8b7a6defa4e8cd5d1fcf864fdf3e5ffa8de61f"
Nov 03 20:30:45.598589 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.598135    5659 scope.go:122] "RemoveContainer" containerID="776d40814d51646d75d9d81f15ce79de12db78d7a867dd6b68205ded835ce2a7"
Nov 03 20:30:45.599048 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:45.598616    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc_odh-ai-gateway-infra(8356069a-b2fa-4a45-935f-9dc18deb9d54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc" podUID="8356069a-b2fa-4a45-935f-9dc18deb9d54"
Nov 03 20:30:45.606237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.606215    5659 scope.go:122] "RemoveContainer" containerID="ff7d07e674fd3e622c2d85ebb06069df5aae590dec26df3439672ea210b2b59f"
Nov 03 20:30:45.606599 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:45.606574    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l_odh-ai-gateway-infra(b88035b6-4c8a-4c03-b803-e897a2c86818)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l" podUID="b88035b6-4c8a-4c03-b803-e897a2c86818"
Nov 03 20:30:45.719073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.718943    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"]
Nov 03 20:30:45.725227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.724920    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:45.925413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.925387    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"
Nov 03 20:30:45.986977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.986869    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"]
Nov 03 20:30:45.988977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.988882    5659 cpu_manager.go:414] "containerMap: removing container" podUID="4729d882-2aaa-4224-ab95-aec46b29a2e7" containerName="cleanup"
Nov 03 20:30:45.988977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.988908    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4729d882-2aaa-4224-ab95-aec46b29a2e7" containerName="cleanup"
Nov 03 20:30:45.988977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.988927    5659 cpu_manager.go:414] "containerMap: removing container" podUID="4729d882-2aaa-4224-ab95-aec46b29a2e7" containerName="cleanup"
Nov 03 20:30:45.988977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.988934    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4729d882-2aaa-4224-ab95-aec46b29a2e7" containerName="cleanup"
Nov 03 20:30:45.989717 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.989625    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="4729d882-2aaa-4224-ab95-aec46b29a2e7" containerName="cleanup"
Nov 03 20:30:45.989717 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.989665    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4069450-679d-44cb-ab4e-44c7fea95e76" containerName="revoke-keys"
Nov 03 20:30:45.989717 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.989682    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="4729d882-2aaa-4224-ab95-aec46b29a2e7" containerName="cleanup"
Nov 03 20:30:45.999181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.999145    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"]
Nov 03 20:30:45.999397 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:45.999378    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"
Nov 03 20:30:46.034111 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.034075    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-kw64c\" (UID: \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"
Nov 03 20:30:46.034263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.034156    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4hq7\" (UniqueName: \"kubernetes.io/projected/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-kube-api-access-g4hq7\") pod \"maas-api-revoke-sub-403cf98d-kw64c\" (UID: \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"
Nov 03 20:30:46.034263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.034232    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-kw64c\" (UID: \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"
Nov 03 20:30:46.135061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.134974    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4729d882-2aaa-4224-ab95-aec46b29a2e7-kube-api-access-8fk64\" (UniqueName: \"kubernetes.io/projected/4729d882-2aaa-4224-ab95-aec46b29a2e7-kube-api-access-8fk64\") pod \"4729d882-2aaa-4224-ab95-aec46b29a2e7\" (UID: \"4729d882-2aaa-4224-ab95-aec46b29a2e7\") "
Nov 03 20:30:46.135303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.135285    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4729d882-2aaa-4224-ab95-aec46b29a2e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4729d882-2aaa-4224-ab95-aec46b29a2e7-maas-api-service-ca\") pod \"4729d882-2aaa-4224-ab95-aec46b29a2e7\" (UID: \"4729d882-2aaa-4224-ab95-aec46b29a2e7\") "
Nov 03 20:30:46.135706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.135675    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4729d882-2aaa-4224-ab95-aec46b29a2e7-maas-api-service-ca" pod "4729d882-2aaa-4224-ab95-aec46b29a2e7" (UID: "4729d882-2aaa-4224-ab95-aec46b29a2e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:46.135871 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.135848    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-kw64c\" (UID: \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"
Nov 03 20:30:46.135947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.135933    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hq7\" (UniqueName: \"kubernetes.io/projected/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-kube-api-access-g4hq7\") pod \"maas-api-revoke-sub-403cf98d-kw64c\" (UID: \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"
Nov 03 20:30:46.136433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.136023    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-kw64c\" (UID: \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"
Nov 03 20:30:46.136433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.136391    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4729d882-2aaa-4224-ab95-aec46b29a2e7-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:46.136661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.136624    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-kw64c\" (UID: \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"
Nov 03 20:30:46.137642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.137610    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4729d882-2aaa-4224-ab95-aec46b29a2e7-kube-api-access-8fk64" pod "4729d882-2aaa-4224-ab95-aec46b29a2e7" (UID: "4729d882-2aaa-4224-ab95-aec46b29a2e7"). InnerVolumeSpecName "kube-api-access-8fk64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:46.138460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.138436    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-kw64c\" (UID: \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"
Nov 03 20:30:46.149103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.149073    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hq7\" (UniqueName: \"kubernetes.io/projected/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-kube-api-access-g4hq7\") pod \"maas-api-revoke-sub-403cf98d-kw64c\" (UID: \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"
Nov 03 20:30:46.238121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.238083    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fk64\" (UniqueName: \"kubernetes.io/projected/4729d882-2aaa-4224-ab95-aec46b29a2e7-kube-api-access-8fk64\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:46.337283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.337244    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"
Nov 03 20:30:46.618024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.617995    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"
Nov 03 20:30:46.618846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.617997    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf" event={"ID":"4729d882-2aaa-4224-ab95-aec46b29a2e7","Type":"ContainerDied","Data":"f7bf78ec2cfefa8a4916cee2a155f743f98921641f329527696e72753b2f4fbe"}
Nov 03 20:30:46.618990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.618912    5659 scope.go:122] "RemoveContainer" containerID="a05c7eb63aef8bac118f05060f6471a4b399049a308c334b63f572d2feb49d46"
Nov 03 20:30:46.656143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.655804    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"]
Nov 03 20:30:46.673155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.671920    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"]
Nov 03 20:30:46.682392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.682295    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88a18e-29895630-rgjlf"]
Nov 03 20:30:46.750531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.750477    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:46.883554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.882396    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"
Nov 03 20:30:46.915577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.915404    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"]
Nov 03 20:30:46.916386 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.916362    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8356069a-b2fa-4a45-935f-9dc18deb9d54" containerName="cleanup"
Nov 03 20:30:46.916386 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.916384    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8356069a-b2fa-4a45-935f-9dc18deb9d54" containerName="cleanup"
Nov 03 20:30:46.916640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.916422    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8356069a-b2fa-4a45-935f-9dc18deb9d54" containerName="cleanup"
Nov 03 20:30:46.916640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.916427    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8356069a-b2fa-4a45-935f-9dc18deb9d54" containerName="cleanup"
Nov 03 20:30:46.916640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.916444    5659 cpu_manager.go:414] "containerMap: removing container" podUID="4729d882-2aaa-4224-ab95-aec46b29a2e7" containerName="cleanup"
Nov 03 20:30:46.916640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.916450    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4729d882-2aaa-4224-ab95-aec46b29a2e7" containerName="cleanup"
Nov 03 20:30:46.916640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.916621    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8356069a-b2fa-4a45-935f-9dc18deb9d54" containerName="cleanup"
Nov 03 20:30:46.916640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.916635    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8356069a-b2fa-4a45-935f-9dc18deb9d54" containerName="cleanup"
Nov 03 20:30:46.916922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.916652    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="4729d882-2aaa-4224-ab95-aec46b29a2e7" containerName="cleanup"
Nov 03 20:30:46.919623 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.919603    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"
Nov 03 20:30:46.937327 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:46.937300    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"]
Nov 03 20:30:47.067270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.067229    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8356069a-b2fa-4a45-935f-9dc18deb9d54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356069a-b2fa-4a45-935f-9dc18deb9d54-maas-api-service-ca\") pod \"8356069a-b2fa-4a45-935f-9dc18deb9d54\" (UID: \"8356069a-b2fa-4a45-935f-9dc18deb9d54\") "
Nov 03 20:30:47.067460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.067390    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8356069a-b2fa-4a45-935f-9dc18deb9d54-kube-api-access-p82jl\" (UniqueName: \"kubernetes.io/projected/8356069a-b2fa-4a45-935f-9dc18deb9d54-kube-api-access-p82jl\") pod \"8356069a-b2fa-4a45-935f-9dc18deb9d54\" (UID: \"8356069a-b2fa-4a45-935f-9dc18deb9d54\") "
Nov 03 20:30:47.067679 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.067655    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8356069a-b2fa-4a45-935f-9dc18deb9d54-maas-api-service-ca" pod "8356069a-b2fa-4a45-935f-9dc18deb9d54" (UID: "8356069a-b2fa-4a45-935f-9dc18deb9d54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:47.067817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.067799    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7b4w\" (UniqueName: \"kubernetes.io/projected/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-kube-api-access-f7b4w\") pod \"maas-api-revoke-sub-5bac6f1e-6z5q8\" (UID: \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"
Nov 03 20:30:47.068062 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.068035    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-6z5q8\" (UID: \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"
Nov 03 20:30:47.068343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.068318    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-6z5q8\" (UID: \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"
Nov 03 20:30:47.068960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.068941    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8356069a-b2fa-4a45-935f-9dc18deb9d54-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:47.069909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.069886    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8356069a-b2fa-4a45-935f-9dc18deb9d54-kube-api-access-p82jl" pod "8356069a-b2fa-4a45-935f-9dc18deb9d54" (UID: "8356069a-b2fa-4a45-935f-9dc18deb9d54"). InnerVolumeSpecName "kube-api-access-p82jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:47.171042 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.170959    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7b4w\" (UniqueName: \"kubernetes.io/projected/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-kube-api-access-f7b4w\") pod \"maas-api-revoke-sub-5bac6f1e-6z5q8\" (UID: \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"
Nov 03 20:30:47.171210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.171087    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-6z5q8\" (UID: \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"
Nov 03 20:30:47.171210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.171158    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-6z5q8\" (UID: \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"
Nov 03 20:30:47.171420 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.171399    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p82jl\" (UniqueName: \"kubernetes.io/projected/8356069a-b2fa-4a45-935f-9dc18deb9d54-kube-api-access-p82jl\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:47.171857 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.171831    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-6z5q8\" (UID: \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"
Nov 03 20:30:47.173645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.173617    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-6z5q8\" (UID: \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"
Nov 03 20:30:47.186784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.186763    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7b4w\" (UniqueName: \"kubernetes.io/projected/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-kube-api-access-f7b4w\") pod \"maas-api-revoke-sub-5bac6f1e-6z5q8\" (UID: \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"
Nov 03 20:30:47.245293 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.245255    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"
Nov 03 20:30:47.542152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.542092    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"]
Nov 03 20:30:47.548306 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:47.548270    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f1d55f_acb8_45e5_bffe_96f1937c0d64.slice/crio-c62793d118a1ff2807351c0a0033f47ad1c6de5eb139358f571afdcece8126b5 WatchSource:0}: Error finding container c62793d118a1ff2807351c0a0033f47ad1c6de5eb139358f571afdcece8126b5: Status 404 returned error can't find the container with id c62793d118a1ff2807351c0a0033f47ad1c6de5eb139358f571afdcece8126b5
Nov 03 20:30:47.597120 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.596820    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:47.707212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.707055    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"
Nov 03 20:30:47.707724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.707054    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc" event={"ID":"8356069a-b2fa-4a45-935f-9dc18deb9d54","Type":"ContainerDied","Data":"f23817924866b233bc838ce4fff4cea22e61a51b2fce2bdb395ef7e881482c71"}
Nov 03 20:30:47.707724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.707317    5659 scope.go:122] "RemoveContainer" containerID="776d40814d51646d75d9d81f15ce79de12db78d7a867dd6b68205ded835ce2a7"
Nov 03 20:30:47.713113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.711895    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c" event={"ID":"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b","Type":"ContainerStarted","Data":"b6232d9381f2300353dd7b7d7f85d0b67e4a6d46dd6721d6dbaa37bd029ef4ab"}
Nov 03 20:30:47.713113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.711955    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c" event={"ID":"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b","Type":"ContainerStarted","Data":"ff933c6135c4e713f295aad091b2ae0353999a8f530a5efc6b978380ba11b9f1"}
Nov 03 20:30:47.715745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.715673    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8" event={"ID":"c6f1d55f-acb8-45e5-bffe-96f1937c0d64","Type":"ContainerStarted","Data":"c62793d118a1ff2807351c0a0033f47ad1c6de5eb139358f571afdcece8126b5"}
Nov 03 20:30:47.738847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.737085    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c" podStartSLOduration=6.737065665 podStartE2EDuration="6.737065665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:47.730727642 +0000 UTC m=+3199.885630062" watchObservedRunningTime="2026-11-03 20:30:47.737065665 +0000 UTC m=+3199.891968085"
Nov 03 20:30:47.749786 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.749743    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"]
Nov 03 20:30:47.759862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.759827    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-98529d-29895630-wdsgc"]
Nov 03 20:30:47.972713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.972678    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4729d882-2aaa-4224-ab95-aec46b29a2e7" path="/var/lib/kubelet/pods/4729d882-2aaa-4224-ab95-aec46b29a2e7/volumes"
Nov 03 20:30:47.973332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:47.973309    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8356069a-b2fa-4a45-935f-9dc18deb9d54" path="/var/lib/kubelet/pods/8356069a-b2fa-4a45-935f-9dc18deb9d54/volumes"
Nov 03 20:30:48.441191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:48.441150    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:48.733626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:48.733591    5659 generic.go:356] "Generic (PLEG): container finished" podID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" containerID="b6232d9381f2300353dd7b7d7f85d0b67e4a6d46dd6721d6dbaa37bd029ef4ab" exitCode=22
Nov 03 20:30:48.734043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:48.733685    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c" event={"ID":"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b","Type":"ContainerDied","Data":"b6232d9381f2300353dd7b7d7f85d0b67e4a6d46dd6721d6dbaa37bd029ef4ab"}
Nov 03 20:30:48.734043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:48.734022    5659 scope.go:122] "RemoveContainer" containerID="b6232d9381f2300353dd7b7d7f85d0b67e4a6d46dd6721d6dbaa37bd029ef4ab"
Nov 03 20:30:48.736236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:48.735711    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8" event={"ID":"c6f1d55f-acb8-45e5-bffe-96f1937c0d64","Type":"ContainerStarted","Data":"ed846ce4b7cc967b2173bac7ede17610bd2b141917550f73eea0dc4ee80a299d"}
Nov 03 20:30:48.777378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:48.777323    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8" podStartSLOduration=6.777308166 podStartE2EDuration="6.777308166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:48.772724431 +0000 UTC m=+3200.927626851" watchObservedRunningTime="2026-11-03 20:30:48.777308166 +0000 UTC m=+3200.932210610"
Nov 03 20:30:49.279502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:49.275732    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:49.752502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:49.750920    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c" event={"ID":"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b","Type":"ContainerStarted","Data":"8a7287031f0bbdfb6c6a89b349cf9d7a37acae17aec29d066d91c7a32bdd338f"}
Nov 03 20:30:49.755532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:49.753310    5659 generic.go:356] "Generic (PLEG): container finished" podID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64" containerID="ed846ce4b7cc967b2173bac7ede17610bd2b141917550f73eea0dc4ee80a299d" exitCode=22
Nov 03 20:30:49.755532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:49.753466    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8" event={"ID":"c6f1d55f-acb8-45e5-bffe-96f1937c0d64","Type":"ContainerDied","Data":"ed846ce4b7cc967b2173bac7ede17610bd2b141917550f73eea0dc4ee80a299d"}
Nov 03 20:30:49.755532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:49.753805    5659 scope.go:122] "RemoveContainer" containerID="ed846ce4b7cc967b2173bac7ede17610bd2b141917550f73eea0dc4ee80a299d"
Nov 03 20:30:49.971618 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:49.971581    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:50.733767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:50.717139    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:50.860174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:50.857462    5659 generic.go:356] "Generic (PLEG): container finished" podID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" containerID="8a7287031f0bbdfb6c6a89b349cf9d7a37acae17aec29d066d91c7a32bdd338f" exitCode=22
Nov 03 20:30:50.860174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:50.857629    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c" event={"ID":"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b","Type":"ContainerDied","Data":"8a7287031f0bbdfb6c6a89b349cf9d7a37acae17aec29d066d91c7a32bdd338f"}
Nov 03 20:30:50.860174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:50.857688    5659 scope.go:122] "RemoveContainer" containerID="b6232d9381f2300353dd7b7d7f85d0b67e4a6d46dd6721d6dbaa37bd029ef4ab"
Nov 03 20:30:50.860174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:50.858230    5659 scope.go:122] "RemoveContainer" containerID="8a7287031f0bbdfb6c6a89b349cf9d7a37acae17aec29d066d91c7a32bdd338f"
Nov 03 20:30:50.862731 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:50.862341    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-kw64c_odh-ai-gateway-infra(20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c" podUID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b"
Nov 03 20:30:50.866435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:50.866405    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8" event={"ID":"c6f1d55f-acb8-45e5-bffe-96f1937c0d64","Type":"ContainerStarted","Data":"58366ccee1839a736011290888bb84541c5b61bdc9fb87c0a637d25c705ea5d3"}
Nov 03 20:30:50.870434 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:50.870407    5659 generic.go:356] "Generic (PLEG): container finished" podID="f421d2e3-252b-45a6-b625-838fa53361c2" containerID="2b95e4f4a0f160d5565ba937a78dafb461bb5961ee8fdd8a0769311d6ec8df93" exitCode=0
Nov 03 20:30:50.870734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:50.870684    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c" event={"ID":"f421d2e3-252b-45a6-b625-838fa53361c2","Type":"ContainerDied","Data":"2b95e4f4a0f160d5565ba937a78dafb461bb5961ee8fdd8a0769311d6ec8df93"}
Nov 03 20:30:50.969897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:50.969859    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:30:51.011877 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.011783    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"]
Nov 03 20:30:51.012716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.012692    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8356069a-b2fa-4a45-935f-9dc18deb9d54" containerName="cleanup"
Nov 03 20:30:51.012716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.012713    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8356069a-b2fa-4a45-935f-9dc18deb9d54" containerName="cleanup"
Nov 03 20:30:51.012934 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.012775    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f421d2e3-252b-45a6-b625-838fa53361c2" containerName="istio-proxy"
Nov 03 20:30:51.012934 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.012783    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f421d2e3-252b-45a6-b625-838fa53361c2" containerName="istio-proxy"
Nov 03 20:30:51.013047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.013017    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f421d2e3-252b-45a6-b625-838fa53361c2" containerName="istio-proxy"
Nov 03 20:30:51.013047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.013043    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8356069a-b2fa-4a45-935f-9dc18deb9d54" containerName="cleanup"
Nov 03 20:30:51.017311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.017290    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"
Nov 03 20:30:51.019159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.019138    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"]
Nov 03 20:30:51.053715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.038665    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"]
Nov 03 20:30:51.053715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.038781    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"
Nov 03 20:30:51.053715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.050186    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"]
Nov 03 20:30:51.053715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.053573    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"
Nov 03 20:30:51.066136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.055801    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"
Nov 03 20:30:51.066136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.058978    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"]
Nov 03 20:30:51.069219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.069196    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"
Nov 03 20:30:51.069576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.069557    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"]
Nov 03 20:30:51.082772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.082748    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"]
Nov 03 20:30:51.084734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.082870    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.152794    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-envoy\") pod \"f421d2e3-252b-45a6-b625-838fa53361c2\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") "
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.154991    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-istio-token\" (UniqueName: \"kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-istio-token\") pod \"f421d2e3-252b-45a6-b625-838fa53361c2\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") "
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.155072    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-socket\") pod \"f421d2e3-252b-45a6-b625-838fa53361c2\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") "
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.155384    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-socket" pod "f421d2e3-252b-45a6-b625-838fa53361c2" (UID: "f421d2e3-252b-45a6-b625-838fa53361c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.155441    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-certs\") pod \"f421d2e3-252b-45a6-b625-838fa53361c2\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") "
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.155706    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-certs" pod "f421d2e3-252b-45a6-b625-838fa53361c2" (UID: "f421d2e3-252b-45a6-b625-838fa53361c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.155788    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-kube-api-access-wgnrs\" (UniqueName: \"kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-kube-api-access-wgnrs\") pod \"f421d2e3-252b-45a6-b625-838fa53361c2\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") "
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.156145    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f421d2e3-252b-45a6-b625-838fa53361c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f421d2e3-252b-45a6-b625-838fa53361c2-istio-podinfo\") pod \"f421d2e3-252b-45a6-b625-838fa53361c2\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") "
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.157432    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f421d2e3-252b-45a6-b625-838fa53361c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f421d2e3-252b-45a6-b625-838fa53361c2-istiod-ca-cert\") pod \"f421d2e3-252b-45a6-b625-838fa53361c2\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") "
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.157696    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-data\") pod \"f421d2e3-252b-45a6-b625-838fa53361c2\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") "
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.157771    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-credential-socket\") pod \"f421d2e3-252b-45a6-b625-838fa53361c2\" (UID: \"f421d2e3-252b-45a6-b625-838fa53361c2\") "
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.159031    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d\" (UID: \"29399fbb-4180-4dcc-b2a0-4a18505e1fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.159165    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs8cf\" (UniqueName: \"kubernetes.io/projected/56bf2d77-4688-4756-a9bb-01f7b346457c-kube-api-access-bs8cf\") pod \"maas-api-revoke-sub-266d0609-tbfnj\" (UID: \"56bf2d77-4688-4756-a9bb-01f7b346457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.159250    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2llk9\" (UniqueName: \"kubernetes.io/projected/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-kube-api-access-2llk9\") pod \"maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d\" (UID: \"29399fbb-4180-4dcc-b2a0-4a18505e1fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.159403    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-tbfnj\" (UID: \"56bf2d77-4688-4756-a9bb-01f7b346457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.159458    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v\" (UID: \"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.159537    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtrl6\" (UniqueName: \"kubernetes.io/projected/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-kube-api-access-jtrl6\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v\" (UID: \"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.159620    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bnsk\" (UniqueName: \"kubernetes.io/projected/208a1108-542a-413d-b038-9b087bb636c8-kube-api-access-8bnsk\") pod \"maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8\" (UID: \"208a1108-542a-413d-b038-9b087bb636c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.159670    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-tbfnj\" (UID: \"56bf2d77-4688-4756-a9bb-01f7b346457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.159778    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/208a1108-542a-413d-b038-9b087bb636c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8\" (UID: \"208a1108-542a-413d-b038-9b087bb636c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.159955    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:51.161514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.159975    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:51.183625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.164726    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f421d2e3-252b-45a6-b625-838fa53361c2-istiod-ca-cert" pod "f421d2e3-252b-45a6-b625-838fa53361c2" (UID: "f421d2e3-252b-45a6-b625-838fa53361c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:51.183625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.164955    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-data" pod "f421d2e3-252b-45a6-b625-838fa53361c2" (UID: "f421d2e3-252b-45a6-b625-838fa53361c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:51.183625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.165278    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-credential-socket" pod "f421d2e3-252b-45a6-b625-838fa53361c2" (UID: "f421d2e3-252b-45a6-b625-838fa53361c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:51.183625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.179264    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-istio-token" pod "f421d2e3-252b-45a6-b625-838fa53361c2" (UID: "f421d2e3-252b-45a6-b625-838fa53361c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:51.183625 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.180774    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f421d2e3-252b-45a6-b625-838fa53361c2-istio-podinfo" pod "f421d2e3-252b-45a6-b625-838fa53361c2" (UID: "f421d2e3-252b-45a6-b625-838fa53361c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:30:51.190784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.185819    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-kube-api-access-wgnrs" pod "f421d2e3-252b-45a6-b625-838fa53361c2" (UID: "f421d2e3-252b-45a6-b625-838fa53361c2"). InnerVolumeSpecName "kube-api-access-wgnrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:51.190784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.188739    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-envoy" pod "f421d2e3-252b-45a6-b625-838fa53361c2" (UID: "f421d2e3-252b-45a6-b625-838fa53361c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:51.194885 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.194284    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"
Nov 03 20:30:51.195740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.195715    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"]
Nov 03 20:30:51.202563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.202429    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"]
Nov 03 20:30:51.202890 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.202769    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"
Nov 03 20:30:51.210940 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.208557    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"]
Nov 03 20:30:51.210940 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.208680    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"
Nov 03 20:30:51.216613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.216593    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"]
Nov 03 20:30:51.216709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.216638    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"]
Nov 03 20:30:51.216709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.216665    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"]
Nov 03 20:30:51.216709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.216691    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"]
Nov 03 20:30:51.216818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.216770    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"
Nov 03 20:30:51.217876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.217858    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"]
Nov 03 20:30:51.243700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.228040    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"]
Nov 03 20:30:51.247434 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.247412    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"]
Nov 03 20:30:51.258661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.258406    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"]
Nov 03 20:30:51.263872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.263231    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59h9\" (UniqueName: \"kubernetes.io/projected/2bc004c1-979c-4f3e-ba19-fddcd5e58629-kube-api-access-r59h9\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc\" (UID: \"2bc004c1-979c-4f3e-ba19-fddcd5e58629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"
Nov 03 20:30:51.263872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.263404    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc004c1-979c-4f3e-ba19-fddcd5e58629-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc\" (UID: \"2bc004c1-979c-4f3e-ba19-fddcd5e58629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"
Nov 03 20:30:51.263872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.263454    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d\" (UID: \"29399fbb-4180-4dcc-b2a0-4a18505e1fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"
Nov 03 20:30:51.263872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.263552    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs8cf\" (UniqueName: \"kubernetes.io/projected/56bf2d77-4688-4756-a9bb-01f7b346457c-kube-api-access-bs8cf\") pod \"maas-api-revoke-sub-266d0609-tbfnj\" (UID: \"56bf2d77-4688-4756-a9bb-01f7b346457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"
Nov 03 20:30:51.263872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.263607    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2llk9\" (UniqueName: \"kubernetes.io/projected/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-kube-api-access-2llk9\") pod \"maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d\" (UID: \"29399fbb-4180-4dcc-b2a0-4a18505e1fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"
Nov 03 20:30:51.263872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.263664    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31961687-b15e-45a0-856a-874eb4baa3ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5\" (UID: \"31961687-b15e-45a0-856a-874eb4baa3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"
Nov 03 20:30:51.263872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.263726    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6t4w\" (UniqueName: \"kubernetes.io/projected/31961687-b15e-45a0-856a-874eb4baa3ec-kube-api-access-p6t4w\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5\" (UID: \"31961687-b15e-45a0-856a-874eb4baa3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"
Nov 03 20:30:51.263872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.263802    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-tbfnj\" (UID: \"56bf2d77-4688-4756-a9bb-01f7b346457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"
Nov 03 20:30:51.265699 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.265669    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v\" (UID: \"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"
Nov 03 20:30:51.265819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.265755    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtrl6\" (UniqueName: \"kubernetes.io/projected/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-kube-api-access-jtrl6\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v\" (UID: \"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"
Nov 03 20:30:51.265887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.265836    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bnsk\" (UniqueName: \"kubernetes.io/projected/208a1108-542a-413d-b038-9b087bb636c8-kube-api-access-8bnsk\") pod \"maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8\" (UID: \"208a1108-542a-413d-b038-9b087bb636c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"
Nov 03 20:30:51.265887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.265867    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-tbfnj\" (UID: \"56bf2d77-4688-4756-a9bb-01f7b346457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"
Nov 03 20:30:51.266008 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.265957    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/208a1108-542a-413d-b038-9b087bb636c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8\" (UID: \"208a1108-542a-413d-b038-9b087bb636c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"
Nov 03 20:30:51.266151 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.266134    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgnrs\" (UniqueName: \"kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-kube-api-access-wgnrs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:51.266270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.266159    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f421d2e3-252b-45a6-b625-838fa53361c2-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:51.266270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.266181    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f421d2e3-252b-45a6-b625-838fa53361c2-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:51.266270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.266196    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:51.266270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.266212    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:51.266270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.266228    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f421d2e3-252b-45a6-b625-838fa53361c2-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:51.266270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.266252    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f421d2e3-252b-45a6-b625-838fa53361c2-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:51.268589 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.268314    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/208a1108-542a-413d-b038-9b087bb636c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8\" (UID: \"208a1108-542a-413d-b038-9b087bb636c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"
Nov 03 20:30:51.269064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.269039    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v\" (UID: \"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"
Nov 03 20:30:51.270921 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.270892    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-tbfnj\" (UID: \"56bf2d77-4688-4756-a9bb-01f7b346457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"
Nov 03 20:30:51.272863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.272835    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d\" (UID: \"29399fbb-4180-4dcc-b2a0-4a18505e1fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"
Nov 03 20:30:51.282452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.282405    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtrl6\" (UniqueName: \"kubernetes.io/projected/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-kube-api-access-jtrl6\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v\" (UID: \"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"
Nov 03 20:30:51.282770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.282749    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-tbfnj\" (UID: \"56bf2d77-4688-4756-a9bb-01f7b346457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"
Nov 03 20:30:51.284925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.284900    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs8cf\" (UniqueName: \"kubernetes.io/projected/56bf2d77-4688-4756-a9bb-01f7b346457c-kube-api-access-bs8cf\") pod \"maas-api-revoke-sub-266d0609-tbfnj\" (UID: \"56bf2d77-4688-4756-a9bb-01f7b346457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"
Nov 03 20:30:51.288057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.288033    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2llk9\" (UniqueName: \"kubernetes.io/projected/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-kube-api-access-2llk9\") pod \"maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d\" (UID: \"29399fbb-4180-4dcc-b2a0-4a18505e1fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"
Nov 03 20:30:51.289890 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.289313    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bnsk\" (UniqueName: \"kubernetes.io/projected/208a1108-542a-413d-b038-9b087bb636c8-kube-api-access-8bnsk\") pod \"maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8\" (UID: \"208a1108-542a-413d-b038-9b087bb636c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"
Nov 03 20:30:51.313501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.313461    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"]
Nov 03 20:30:51.324366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.324338    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"]
Nov 03 20:30:51.367638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.367590    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18783ab-02f5-4154-bc33-bc738babbd40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks\" (UID: \"f18783ab-02f5-4154-bc33-bc738babbd40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"
Nov 03 20:30:51.367827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.367654    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a370c44d-12fc-4874-a663-70e1b2fd44ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj\" (UID: \"a370c44d-12fc-4874-a663-70e1b2fd44ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"
Nov 03 20:30:51.367827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.367805    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc004c1-979c-4f3e-ba19-fddcd5e58629-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc\" (UID: \"2bc004c1-979c-4f3e-ba19-fddcd5e58629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"
Nov 03 20:30:51.368004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.367986    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc5e85b-47e2-44f9-990c-7cd7780e3256-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-kn9fh\" (UID: \"bcc5e85b-47e2-44f9-990c-7cd7780e3256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"
Nov 03 20:30:51.368084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.368025    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31961687-b15e-45a0-856a-874eb4baa3ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5\" (UID: \"31961687-b15e-45a0-856a-874eb4baa3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"
Nov 03 20:30:51.368146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.368099    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gtfc\" (UniqueName: \"kubernetes.io/projected/f938b051-c9cc-4600-89b3-f2a4fde514ec-kube-api-access-8gtfc\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc\" (UID: \"f938b051-c9cc-4600-89b3-f2a4fde514ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"
Nov 03 20:30:51.368146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.368132    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xz9\" (UniqueName: \"kubernetes.io/projected/a370c44d-12fc-4874-a663-70e1b2fd44ad-kube-api-access-b2xz9\") pod \"maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj\" (UID: \"a370c44d-12fc-4874-a663-70e1b2fd44ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"
Nov 03 20:30:51.368260 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.368165    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f938b051-c9cc-4600-89b3-f2a4fde514ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc\" (UID: \"f938b051-c9cc-4600-89b3-f2a4fde514ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"
Nov 03 20:30:51.368260 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.368195    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6t4w\" (UniqueName: \"kubernetes.io/projected/31961687-b15e-45a0-856a-874eb4baa3ec-kube-api-access-p6t4w\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5\" (UID: \"31961687-b15e-45a0-856a-874eb4baa3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"
Nov 03 20:30:51.368644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.368618    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc004c1-979c-4f3e-ba19-fddcd5e58629-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc\" (UID: \"2bc004c1-979c-4f3e-ba19-fddcd5e58629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"
Nov 03 20:30:51.368766 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.368682    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgp45\" (UniqueName: \"kubernetes.io/projected/bcc5e85b-47e2-44f9-990c-7cd7780e3256-kube-api-access-fgp45\") pod \"maas-api-key-cleanup-29895630-kn9fh\" (UID: \"bcc5e85b-47e2-44f9-990c-7cd7780e3256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"
Nov 03 20:30:51.368766 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.368720    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59h9\" (UniqueName: \"kubernetes.io/projected/2bc004c1-979c-4f3e-ba19-fddcd5e58629-kube-api-access-r59h9\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc\" (UID: \"2bc004c1-979c-4f3e-ba19-fddcd5e58629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"
Nov 03 20:30:51.368845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.368780    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk27q\" (UniqueName: \"kubernetes.io/projected/f18783ab-02f5-4154-bc33-bc738babbd40-kube-api-access-nk27q\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks\" (UID: \"f18783ab-02f5-4154-bc33-bc738babbd40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"
Nov 03 20:30:51.369175 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.369152    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31961687-b15e-45a0-856a-874eb4baa3ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5\" (UID: \"31961687-b15e-45a0-856a-874eb4baa3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"
Nov 03 20:30:51.380550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.380527    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6t4w\" (UniqueName: \"kubernetes.io/projected/31961687-b15e-45a0-856a-874eb4baa3ec-kube-api-access-p6t4w\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5\" (UID: \"31961687-b15e-45a0-856a-874eb4baa3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"
Nov 03 20:30:51.380550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.380544    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59h9\" (UniqueName: \"kubernetes.io/projected/2bc004c1-979c-4f3e-ba19-fddcd5e58629-kube-api-access-r59h9\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc\" (UID: \"2bc004c1-979c-4f3e-ba19-fddcd5e58629\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"
Nov 03 20:30:51.452685 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.452652    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"
Nov 03 20:30:51.470837 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.470799    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc5e85b-47e2-44f9-990c-7cd7780e3256-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-kn9fh\" (UID: \"bcc5e85b-47e2-44f9-990c-7cd7780e3256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"
Nov 03 20:30:51.470973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.470891    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gtfc\" (UniqueName: \"kubernetes.io/projected/f938b051-c9cc-4600-89b3-f2a4fde514ec-kube-api-access-8gtfc\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc\" (UID: \"f938b051-c9cc-4600-89b3-f2a4fde514ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"
Nov 03 20:30:51.470973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.470926    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xz9\" (UniqueName: \"kubernetes.io/projected/a370c44d-12fc-4874-a663-70e1b2fd44ad-kube-api-access-b2xz9\") pod \"maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj\" (UID: \"a370c44d-12fc-4874-a663-70e1b2fd44ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"
Nov 03 20:30:51.471179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.471148    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f938b051-c9cc-4600-89b3-f2a4fde514ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc\" (UID: \"f938b051-c9cc-4600-89b3-f2a4fde514ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"
Nov 03 20:30:51.471606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.471575    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgp45\" (UniqueName: \"kubernetes.io/projected/bcc5e85b-47e2-44f9-990c-7cd7780e3256-kube-api-access-fgp45\") pod \"maas-api-key-cleanup-29895630-kn9fh\" (UID: \"bcc5e85b-47e2-44f9-990c-7cd7780e3256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"
Nov 03 20:30:51.471712 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.471652    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc5e85b-47e2-44f9-990c-7cd7780e3256-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-kn9fh\" (UID: \"bcc5e85b-47e2-44f9-990c-7cd7780e3256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"
Nov 03 20:30:51.471712 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.471679    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk27q\" (UniqueName: \"kubernetes.io/projected/f18783ab-02f5-4154-bc33-bc738babbd40-kube-api-access-nk27q\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks\" (UID: \"f18783ab-02f5-4154-bc33-bc738babbd40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"
Nov 03 20:30:51.471847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.471753    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18783ab-02f5-4154-bc33-bc738babbd40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks\" (UID: \"f18783ab-02f5-4154-bc33-bc738babbd40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"
Nov 03 20:30:51.471847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.471758    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f938b051-c9cc-4600-89b3-f2a4fde514ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc\" (UID: \"f938b051-c9cc-4600-89b3-f2a4fde514ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"
Nov 03 20:30:51.471847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.471790    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a370c44d-12fc-4874-a663-70e1b2fd44ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj\" (UID: \"a370c44d-12fc-4874-a663-70e1b2fd44ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"
Nov 03 20:30:51.472673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.472651    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a370c44d-12fc-4874-a663-70e1b2fd44ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj\" (UID: \"a370c44d-12fc-4874-a663-70e1b2fd44ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"
Nov 03 20:30:51.472827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.472793    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18783ab-02f5-4154-bc33-bc738babbd40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks\" (UID: \"f18783ab-02f5-4154-bc33-bc738babbd40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"
Nov 03 20:30:51.481243 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.481220    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"
Nov 03 20:30:51.485179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.485153    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgp45\" (UniqueName: \"kubernetes.io/projected/bcc5e85b-47e2-44f9-990c-7cd7780e3256-kube-api-access-fgp45\") pod \"maas-api-key-cleanup-29895630-kn9fh\" (UID: \"bcc5e85b-47e2-44f9-990c-7cd7780e3256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"
Nov 03 20:30:51.488515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.485309    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gtfc\" (UniqueName: \"kubernetes.io/projected/f938b051-c9cc-4600-89b3-f2a4fde514ec-kube-api-access-8gtfc\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc\" (UID: \"f938b051-c9cc-4600-89b3-f2a4fde514ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"
Nov 03 20:30:51.488515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.486064    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk27q\" (UniqueName: \"kubernetes.io/projected/f18783ab-02f5-4154-bc33-bc738babbd40-kube-api-access-nk27q\") pod \"maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks\" (UID: \"f18783ab-02f5-4154-bc33-bc738babbd40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"
Nov 03 20:30:51.488515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.486565    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xz9\" (UniqueName: \"kubernetes.io/projected/a370c44d-12fc-4874-a663-70e1b2fd44ad-kube-api-access-b2xz9\") pod \"maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj\" (UID: \"a370c44d-12fc-4874-a663-70e1b2fd44ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"
Nov 03 20:30:51.569092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.568803    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"
Nov 03 20:30:51.587343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.586982    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"
Nov 03 20:30:51.596195 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.596066    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:51.598741 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.598516    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"
Nov 03 20:30:51.630332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.629907    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"
Nov 03 20:30:51.656264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.647844    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"
Nov 03 20:30:51.656264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.648002    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"
Nov 03 20:30:51.656264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.649007    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"
Nov 03 20:30:51.656264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.655715    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"
Nov 03 20:30:51.935797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.934827    5659 scope.go:122] "RemoveContainer" containerID="8a7287031f0bbdfb6c6a89b349cf9d7a37acae17aec29d066d91c7a32bdd338f"
Nov 03 20:30:51.935797 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:51.935227    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-kw64c_odh-ai-gateway-infra(20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c" podUID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b"
Nov 03 20:30:51.958090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.955380    5659 generic.go:356] "Generic (PLEG): container finished" podID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64" containerID="58366ccee1839a736011290888bb84541c5b61bdc9fb87c0a637d25c705ea5d3" exitCode=22
Nov 03 20:30:51.958090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.955537    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8" event={"ID":"c6f1d55f-acb8-45e5-bffe-96f1937c0d64","Type":"ContainerDied","Data":"58366ccee1839a736011290888bb84541c5b61bdc9fb87c0a637d25c705ea5d3"}
Nov 03 20:30:51.958090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.955598    5659 scope.go:122] "RemoveContainer" containerID="ed846ce4b7cc967b2173bac7ede17610bd2b141917550f73eea0dc4ee80a299d"
Nov 03 20:30:51.958090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.956328    5659 scope.go:122] "RemoveContainer" containerID="58366ccee1839a736011290888bb84541c5b61bdc9fb87c0a637d25c705ea5d3"
Nov 03 20:30:51.958090 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:51.956798    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-6z5q8_odh-ai-gateway-infra(c6f1d55f-acb8-45e5-bffe-96f1937c0d64)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8" podUID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64"
Nov 03 20:30:51.965694 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:51.965664    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56bf2d77_4688_4756_a9bb_01f7b346457c.slice/crio-34280dce94d696ee4f70bef2bab56b2ca1364cad328abf298c0d0f2b9b10fa15 WatchSource:0}: Error finding container 34280dce94d696ee4f70bef2bab56b2ca1364cad328abf298c0d0f2b9b10fa15: Status 404 returned error can't find the container with id 34280dce94d696ee4f70bef2bab56b2ca1364cad328abf298c0d0f2b9b10fa15
Nov 03 20:30:51.982696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:51.979962    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"
Nov 03 20:30:52.196409 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:52.185353    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c13e2c_aeb0_4eb2_90ed_78c91e40280e.slice/crio-a35d564c16229896b830f804bcdaae4a510ada1caed29fdfd8dbc22854fcd53b WatchSource:0}: Error finding container a35d564c16229896b830f804bcdaae4a510ada1caed29fdfd8dbc22854fcd53b: Status 404 returned error can't find the container with id a35d564c16229896b830f804bcdaae4a510ada1caed29fdfd8dbc22854fcd53b
Nov 03 20:30:52.196409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.190160    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c" event={"ID":"f421d2e3-252b-45a6-b625-838fa53361c2","Type":"ContainerDied","Data":"c847af84775615425e429b3566524fa2f326e58010efcc36411836ffe46550ac"}
Nov 03 20:30:52.196409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.190293    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"]
Nov 03 20:30:52.196409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.192338    5659 scope.go:122] "RemoveContainer" containerID="2b95e4f4a0f160d5565ba937a78dafb461bb5961ee8fdd8a0769311d6ec8df93"
Nov 03 20:30:52.196409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.193772    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"]
Nov 03 20:30:52.196409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.193823    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"]
Nov 03 20:30:52.196409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.193854    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"]
Nov 03 20:30:52.207431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.205659    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-4269c"]
Nov 03 20:30:52.284382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.284287    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:52.413283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.413229    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"]
Nov 03 20:30:52.419253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.419220    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"]
Nov 03 20:30:52.439531 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:52.437769    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc004c1_979c_4f3e_ba19_fddcd5e58629.slice/crio-7652a14b23cd7db884deb4a5e8674fe6876ccc684c3448e360087865f1704932 WatchSource:0}: Error finding container 7652a14b23cd7db884deb4a5e8674fe6876ccc684c3448e360087865f1704932: Status 404 returned error can't find the container with id 7652a14b23cd7db884deb4a5e8674fe6876ccc684c3448e360087865f1704932
Nov 03 20:30:52.440334 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:52.440309    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208a1108_542a_413d_b038_9b087bb636c8.slice/crio-ac43007cf64c848cebfd3d95d33a8b2ae1ab7c63fe61d768859c8a26d26323e2 WatchSource:0}: Error finding container ac43007cf64c848cebfd3d95d33a8b2ae1ab7c63fe61d768859c8a26d26323e2: Status 404 returned error can't find the container with id ac43007cf64c848cebfd3d95d33a8b2ae1ab7c63fe61d768859c8a26d26323e2
Nov 03 20:30:52.673793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.673758    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"]
Nov 03 20:30:52.678986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.678959    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"]
Nov 03 20:30:52.716366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.716337    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"]
Nov 03 20:30:52.721207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.721180    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"]
Nov 03 20:30:52.730713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.730678    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"]
Nov 03 20:30:52.741436 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:52.741410    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda370c44d_12fc_4874_a663_70e1b2fd44ad.slice/crio-799bcc944e9091f0320f596043bf2b2485b1e393fc07539b831d43ee8bce743e WatchSource:0}: Error finding container 799bcc944e9091f0320f596043bf2b2485b1e393fc07539b831d43ee8bce743e: Status 404 returned error can't find the container with id 799bcc944e9091f0320f596043bf2b2485b1e393fc07539b831d43ee8bce743e
Nov 03 20:30:52.993120 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.993083    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj" event={"ID":"56bf2d77-4688-4756-a9bb-01f7b346457c","Type":"ContainerStarted","Data":"e51baa26be341530efd3541b8d76a60d5d44383684b31e7bc17a2c9a6d6ef3dd"}
Nov 03 20:30:52.993614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.993148    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj" event={"ID":"56bf2d77-4688-4756-a9bb-01f7b346457c","Type":"ContainerStarted","Data":"34280dce94d696ee4f70bef2bab56b2ca1364cad328abf298c0d0f2b9b10fa15"}
Nov 03 20:30:52.994528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.994464    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh" event={"ID":"bcc5e85b-47e2-44f9-990c-7cd7780e3256","Type":"ContainerStarted","Data":"a96fbe5a17eb9d4fe19a08ca9884463832d4daacd6b3968fc1338d78a2bef2d6"}
Nov 03 20:30:52.995795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.995775    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc" event={"ID":"f938b051-c9cc-4600-89b3-f2a4fde514ec","Type":"ContainerStarted","Data":"5caf6e37eb0d159f2fd2125ea473e4937790bacca73bb7a8d7c07f72ebc1b463"}
Nov 03 20:30:52.997343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.997316    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc" event={"ID":"2bc004c1-979c-4f3e-ba19-fddcd5e58629","Type":"ContainerStarted","Data":"7652a14b23cd7db884deb4a5e8674fe6876ccc684c3448e360087865f1704932"}
Nov 03 20:30:52.998709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:52.998686    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v" event={"ID":"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e","Type":"ContainerStarted","Data":"a35d564c16229896b830f804bcdaae4a510ada1caed29fdfd8dbc22854fcd53b"}
Nov 03 20:30:53.000472 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:53.000450    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks" event={"ID":"f18783ab-02f5-4154-bc33-bc738babbd40","Type":"ContainerStarted","Data":"f5906459ea21e64ca99b7fca357cc9759e4fa18a370c377ffcbd18dfef042f85"}
Nov 03 20:30:53.001737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:53.001715    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj" event={"ID":"a370c44d-12fc-4874-a663-70e1b2fd44ad","Type":"ContainerStarted","Data":"799bcc944e9091f0320f596043bf2b2485b1e393fc07539b831d43ee8bce743e"}
Nov 03 20:30:53.003610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:53.003588    5659 scope.go:122] "RemoveContainer" containerID="58366ccee1839a736011290888bb84541c5b61bdc9fb87c0a637d25c705ea5d3"
Nov 03 20:30:53.003890 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:53.003866    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-6z5q8_odh-ai-gateway-infra(c6f1d55f-acb8-45e5-bffe-96f1937c0d64)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8" podUID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64"
Nov 03 20:30:53.004912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:53.004884    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d" event={"ID":"29399fbb-4180-4dcc-b2a0-4a18505e1fb9","Type":"ContainerStarted","Data":"57d09492dc416b3953e3cc607fbd35003563df74d1cf21fe0762b5bf6e2316da"}
Nov 03 20:30:53.005014 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:53.004928    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d" event={"ID":"29399fbb-4180-4dcc-b2a0-4a18505e1fb9","Type":"ContainerStarted","Data":"4e32cfc9f26d151209180016214dc98cf28428a2c765bcd6142f307381d08354"}
Nov 03 20:30:53.007184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:53.007164    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8" event={"ID":"208a1108-542a-413d-b038-9b087bb636c8","Type":"ContainerStarted","Data":"ac43007cf64c848cebfd3d95d33a8b2ae1ab7c63fe61d768859c8a26d26323e2"}
Nov 03 20:30:53.008446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:53.008426    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5" event={"ID":"31961687-b15e-45a0-856a-874eb4baa3ec","Type":"ContainerStarted","Data":"e4098e3e66049b612b54000faaf03366b5d48f250cef3266a6ca3910f3ac4257"}
Nov 03 20:30:53.016566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:53.016527    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj" podStartSLOduration=4.016512051 podStartE2EDuration="4.016512051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:53.01287536 +0000 UTC m=+3205.167777799" watchObservedRunningTime="2026-11-03 20:30:53.016512051 +0000 UTC m=+3205.171414472"
Nov 03 20:30:53.040063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:53.039997    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d" podStartSLOduration=53.039984015 podStartE2EDuration="53.039984015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:53.029980561 +0000 UTC m=+3205.184882981" watchObservedRunningTime="2026-11-03 20:30:53.039984015 +0000 UTC m=+3205.194886472"
Nov 03 20:30:53.313209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:53.313177    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:53.978206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:53.978165    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f421d2e3-252b-45a6-b625-838fa53361c2" path="/var/lib/kubelet/pods/f421d2e3-252b-45a6-b625-838fa53361c2/volumes"
Nov 03 20:30:53.979545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:53.979511    5659 scope.go:122] "RemoveContainer" containerID="10552455d97167d331ad974dfc9f03db46c2f580a755803f5fde776f9550e6b5"
Nov 03 20:30:54.019797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.019763    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc" event={"ID":"f938b051-c9cc-4600-89b3-f2a4fde514ec","Type":"ContainerStarted","Data":"2e89c6c14cf0a704a857fecd1d736f111fe590801b74f81adb9e51e75d467eb2"}
Nov 03 20:30:54.021534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.021468    5659 generic.go:356] "Generic (PLEG): container finished" podID="2bc004c1-979c-4f3e-ba19-fddcd5e58629" containerID="436d0951862364b36747a9d6fc727c57e358187ffc9eb772bd90c4baab38c262" exitCode=60
Nov 03 20:30:54.021666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.021510    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc" event={"ID":"2bc004c1-979c-4f3e-ba19-fddcd5e58629","Type":"ContainerDied","Data":"436d0951862364b36747a9d6fc727c57e358187ffc9eb772bd90c4baab38c262"}
Nov 03 20:30:54.021846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.021825    5659 scope.go:122] "RemoveContainer" containerID="436d0951862364b36747a9d6fc727c57e358187ffc9eb772bd90c4baab38c262"
Nov 03 20:30:54.023422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.023393    5659 generic.go:356] "Generic (PLEG): container finished" podID="c1c13e2c-aeb0-4eb2-90ed-78c91e40280e" containerID="57bf09f061e28ed3a5c69fd06b70ebbdd08ff2d4c447bc9c11bda1873d31c1af" exitCode=60
Nov 03 20:30:54.023555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.023476    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v" event={"ID":"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e","Type":"ContainerDied","Data":"57bf09f061e28ed3a5c69fd06b70ebbdd08ff2d4c447bc9c11bda1873d31c1af"}
Nov 03 20:30:54.023726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.023708    5659 scope.go:122] "RemoveContainer" containerID="57bf09f061e28ed3a5c69fd06b70ebbdd08ff2d4c447bc9c11bda1873d31c1af"
Nov 03 20:30:54.025656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.025464    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks" event={"ID":"f18783ab-02f5-4154-bc33-bc738babbd40","Type":"ContainerStarted","Data":"35c900483b4a6d3fa8b223d646ff00ad132d2f00e17d2176bab0fe139c80ea35"}
Nov 03 20:30:54.027561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.027537    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj" event={"ID":"a370c44d-12fc-4874-a663-70e1b2fd44ad","Type":"ContainerStarted","Data":"b579c803c5fe631d9c0c211f62f053000cb13fd93509dfe1b638826297c62db4"}
Nov 03 20:30:54.029264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.029243    5659 generic.go:356] "Generic (PLEG): container finished" podID="29399fbb-4180-4dcc-b2a0-4a18505e1fb9" containerID="57d09492dc416b3953e3cc607fbd35003563df74d1cf21fe0762b5bf6e2316da" exitCode=60
Nov 03 20:30:54.029367 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.029323    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d" event={"ID":"29399fbb-4180-4dcc-b2a0-4a18505e1fb9","Type":"ContainerDied","Data":"57d09492dc416b3953e3cc607fbd35003563df74d1cf21fe0762b5bf6e2316da"}
Nov 03 20:30:54.029547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.029531    5659 scope.go:122] "RemoveContainer" containerID="57d09492dc416b3953e3cc607fbd35003563df74d1cf21fe0762b5bf6e2316da"
Nov 03 20:30:54.031045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.031021    5659 generic.go:356] "Generic (PLEG): container finished" podID="208a1108-542a-413d-b038-9b087bb636c8" containerID="abfbef523a1ff7b63c1b5227496e7bae8babc2bca5451bab96dbfc3ade2c97c4" exitCode=60
Nov 03 20:30:54.031149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.031099    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8" event={"ID":"208a1108-542a-413d-b038-9b087bb636c8","Type":"ContainerDied","Data":"abfbef523a1ff7b63c1b5227496e7bae8babc2bca5451bab96dbfc3ade2c97c4"}
Nov 03 20:30:54.031325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.031307    5659 scope.go:122] "RemoveContainer" containerID="abfbef523a1ff7b63c1b5227496e7bae8babc2bca5451bab96dbfc3ade2c97c4"
Nov 03 20:30:54.062555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.042188    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc" podStartSLOduration=54.042170123 podStartE2EDuration="54.042170123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:54.040118818 +0000 UTC m=+3206.195021237" watchObservedRunningTime="2026-11-03 20:30:54.042170123 +0000 UTC m=+3206.197072544"
Nov 03 20:30:54.143012 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.141814    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:54.166528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.166085    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5" event={"ID":"31961687-b15e-45a0-856a-874eb4baa3ec","Type":"ContainerStarted","Data":"34910a865d025d602b2a18da8434e16353333987801bbc020cefb378bf6f54f7"}
Nov 03 20:30:54.167344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.166843    5659 scope.go:122] "RemoveContainer" containerID="34910a865d025d602b2a18da8434e16353333987801bbc020cefb378bf6f54f7"
Nov 03 20:30:54.169288 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.169264    5659 generic.go:356] "Generic (PLEG): container finished" podID="56bf2d77-4688-4756-a9bb-01f7b346457c" containerID="e51baa26be341530efd3541b8d76a60d5d44383684b31e7bc17a2c9a6d6ef3dd" exitCode=22
Nov 03 20:30:54.169371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.169333    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj" event={"ID":"56bf2d77-4688-4756-a9bb-01f7b346457c","Type":"ContainerDied","Data":"e51baa26be341530efd3541b8d76a60d5d44383684b31e7bc17a2c9a6d6ef3dd"}
Nov 03 20:30:54.169750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.169731    5659 scope.go:122] "RemoveContainer" containerID="e51baa26be341530efd3541b8d76a60d5d44383684b31e7bc17a2c9a6d6ef3dd"
Nov 03 20:30:54.177526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.176811    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh" event={"ID":"bcc5e85b-47e2-44f9-990c-7cd7780e3256","Type":"ContainerStarted","Data":"44bb1e8713cf1cee7b57284e64d367cec45fa1e4d12d71e180f7fd1bb9fafe1d"}
Nov 03 20:30:54.214877 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.214209    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks" podStartSLOduration=54.214190821 podStartE2EDuration="54.214190821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:54.161950021 +0000 UTC m=+3206.316852465" watchObservedRunningTime="2026-11-03 20:30:54.214190821 +0000 UTC m=+3206.369093241"
Nov 03 20:30:54.226660 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.226599    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj" podStartSLOduration=54.226581513 podStartE2EDuration="54.226581513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:54.206908451 +0000 UTC m=+3206.361810881" watchObservedRunningTime="2026-11-03 20:30:54.226581513 +0000 UTC m=+3206.381483932"
Nov 03 20:30:54.293072 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.293018    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh" podStartSLOduration=54.293003795 podStartE2EDuration="54.293003795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:54.291508478 +0000 UTC m=+3206.446410896" watchObservedRunningTime="2026-11-03 20:30:54.293003795 +0000 UTC m=+3206.447906214"
Nov 03 20:30:54.854546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.854128    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:54.969525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:54.962911    5659 scope.go:122] "RemoveContainer" containerID="884d016e21ff67955fb3b9ea6fe77f1d56ab897fdf56a36fde75ea3fd72a60ff"
Nov 03 20:30:55.204503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.204113    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z" event={"ID":"a15e208a-f352-452c-b139-9526f846d5ab","Type":"ContainerStarted","Data":"40f69d0dfac546b848125fe0f46ee0a91147cf9e63ee7a69de362282ea4ce63e"}
Nov 03 20:30:55.208437 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.208406    5659 generic.go:356] "Generic (PLEG): container finished" podID="f18783ab-02f5-4154-bc33-bc738babbd40" containerID="35c900483b4a6d3fa8b223d646ff00ad132d2f00e17d2176bab0fe139c80ea35" exitCode=60
Nov 03 20:30:55.208736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.208715    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks" event={"ID":"f18783ab-02f5-4154-bc33-bc738babbd40","Type":"ContainerDied","Data":"35c900483b4a6d3fa8b223d646ff00ad132d2f00e17d2176bab0fe139c80ea35"}
Nov 03 20:30:55.209807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.209773    5659 scope.go:122] "RemoveContainer" containerID="35c900483b4a6d3fa8b223d646ff00ad132d2f00e17d2176bab0fe139c80ea35"
Nov 03 20:30:55.224716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.223949    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj" event={"ID":"a370c44d-12fc-4874-a663-70e1b2fd44ad","Type":"ContainerDied","Data":"b579c803c5fe631d9c0c211f62f053000cb13fd93509dfe1b638826297c62db4"}
Nov 03 20:30:55.224716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.224189    5659 scope.go:122] "RemoveContainer" containerID="b579c803c5fe631d9c0c211f62f053000cb13fd93509dfe1b638826297c62db4"
Nov 03 20:30:55.249211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.223848    5659 generic.go:356] "Generic (PLEG): container finished" podID="a370c44d-12fc-4874-a663-70e1b2fd44ad" containerID="b579c803c5fe631d9c0c211f62f053000cb13fd93509dfe1b638826297c62db4" exitCode=60
Nov 03 20:30:55.275915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.275880    5659 generic.go:356] "Generic (PLEG): container finished" podID="31961687-b15e-45a0-856a-874eb4baa3ec" containerID="34910a865d025d602b2a18da8434e16353333987801bbc020cefb378bf6f54f7" exitCode=60
Nov 03 20:30:55.276603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.276569    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5" event={"ID":"31961687-b15e-45a0-856a-874eb4baa3ec","Type":"ContainerDied","Data":"34910a865d025d602b2a18da8434e16353333987801bbc020cefb378bf6f54f7"}
Nov 03 20:30:55.285737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.285695    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj" event={"ID":"56bf2d77-4688-4756-a9bb-01f7b346457c","Type":"ContainerStarted","Data":"07ac5c968ed7900c1dc38b78c7f3ee5783c27e8e0b752dbba411fcfa4e85e08f"}
Nov 03 20:30:55.289098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.289066    5659 generic.go:356] "Generic (PLEG): container finished" podID="bcc5e85b-47e2-44f9-990c-7cd7780e3256" containerID="44bb1e8713cf1cee7b57284e64d367cec45fa1e4d12d71e180f7fd1bb9fafe1d" exitCode=60
Nov 03 20:30:55.289688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.289633    5659 scope.go:122] "RemoveContainer" containerID="44bb1e8713cf1cee7b57284e64d367cec45fa1e4d12d71e180f7fd1bb9fafe1d"
Nov 03 20:30:55.290212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.290185    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh" event={"ID":"bcc5e85b-47e2-44f9-990c-7cd7780e3256","Type":"ContainerDied","Data":"44bb1e8713cf1cee7b57284e64d367cec45fa1e4d12d71e180f7fd1bb9fafe1d"}
Nov 03 20:30:55.292338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.292311    5659 generic.go:356] "Generic (PLEG): container finished" podID="f938b051-c9cc-4600-89b3-f2a4fde514ec" containerID="2e89c6c14cf0a704a857fecd1d736f111fe590801b74f81adb9e51e75d467eb2" exitCode=60
Nov 03 20:30:55.292435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.292367    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc" event={"ID":"f938b051-c9cc-4600-89b3-f2a4fde514ec","Type":"ContainerDied","Data":"2e89c6c14cf0a704a857fecd1d736f111fe590801b74f81adb9e51e75d467eb2"}
Nov 03 20:30:55.292780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.292761    5659 scope.go:122] "RemoveContainer" containerID="2e89c6c14cf0a704a857fecd1d736f111fe590801b74f81adb9e51e75d467eb2"
Nov 03 20:30:55.388236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.388180    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5" podStartSLOduration=55.388160575 podStartE2EDuration="55.388160575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:55.386550377 +0000 UTC m=+3207.541452801" watchObservedRunningTime="2026-11-03 20:30:55.388160575 +0000 UTC m=+3207.543062997"
Nov 03 20:30:55.863539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.863466    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:55.976632 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:55.974169    5659 scope.go:122] "RemoveContainer" containerID="ef4a0fdfc002af3edda96ca26de19c5cc2e29d2e3acc68f1b0ae803fa533223d"
Nov 03 20:30:56.253771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.253663    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"]
Nov 03 20:30:56.325143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.325106    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc" event={"ID":"f938b051-c9cc-4600-89b3-f2a4fde514ec","Type":"ContainerStarted","Data":"ef8dcbbb40cad41f8fd0f1ea5429ccdd1ad574ac875260905b06c62b68ca2790"}
Nov 03 20:30:56.327799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.327763    5659 generic.go:356] "Generic (PLEG): container finished" podID="2bc004c1-979c-4f3e-ba19-fddcd5e58629" containerID="e83f41bcae8b8894f2c9e53468d1c5eb8a6ffca9108f56551d22e419c0c0a3cc" exitCode=60
Nov 03 20:30:56.327799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.327852    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc" event={"ID":"2bc004c1-979c-4f3e-ba19-fddcd5e58629","Type":"ContainerDied","Data":"e83f41bcae8b8894f2c9e53468d1c5eb8a6ffca9108f56551d22e419c0c0a3cc"}
Nov 03 20:30:56.328131 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.327938    5659 scope.go:122] "RemoveContainer" containerID="436d0951862364b36747a9d6fc727c57e358187ffc9eb772bd90c4baab38c262"
Nov 03 20:30:56.328334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.328313    5659 scope.go:122] "RemoveContainer" containerID="e83f41bcae8b8894f2c9e53468d1c5eb8a6ffca9108f56551d22e419c0c0a3cc"
Nov 03 20:30:56.329011 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:56.328984    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc_odh-ai-gateway-infra(2bc004c1-979c-4f3e-ba19-fddcd5e58629)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc" podUID="2bc004c1-979c-4f3e-ba19-fddcd5e58629"
Nov 03 20:30:56.330512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.330455    5659 generic.go:356] "Generic (PLEG): container finished" podID="c1c13e2c-aeb0-4eb2-90ed-78c91e40280e" containerID="1d12f1d96bca9eb3f7e90b44cd41a47d2e52331b1dadbd7aa26ed05a077a9742" exitCode=60
Nov 03 20:30:56.330627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.330544    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v" event={"ID":"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e","Type":"ContainerDied","Data":"1d12f1d96bca9eb3f7e90b44cd41a47d2e52331b1dadbd7aa26ed05a077a9742"}
Nov 03 20:30:56.330927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.330907    5659 scope.go:122] "RemoveContainer" containerID="1d12f1d96bca9eb3f7e90b44cd41a47d2e52331b1dadbd7aa26ed05a077a9742"
Nov 03 20:30:56.331378 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:56.331355    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v_odh-ai-gateway-infra(c1c13e2c-aeb0-4eb2-90ed-78c91e40280e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v" podUID="c1c13e2c-aeb0-4eb2-90ed-78c91e40280e"
Nov 03 20:30:56.387054 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.387017    5659 generic.go:356] "Generic (PLEG): container finished" podID="a15e208a-f352-452c-b139-9526f846d5ab" containerID="40f69d0dfac546b848125fe0f46ee0a91147cf9e63ee7a69de362282ea4ce63e" exitCode=60
Nov 03 20:30:56.388086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.388053    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z" event={"ID":"a15e208a-f352-452c-b139-9526f846d5ab","Type":"ContainerDied","Data":"40f69d0dfac546b848125fe0f46ee0a91147cf9e63ee7a69de362282ea4ce63e"}
Nov 03 20:30:56.418092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.418061    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks" event={"ID":"f18783ab-02f5-4154-bc33-bc738babbd40","Type":"ContainerStarted","Data":"dad3ee195739fad2412a2832a8ae3ca6a545960f3bbf7ef8025dafee17444b38"}
Nov 03 20:30:56.420558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.420406    5659 scope.go:122] "RemoveContainer" containerID="57bf09f061e28ed3a5c69fd06b70ebbdd08ff2d4c447bc9c11bda1873d31c1af"
Nov 03 20:30:56.423475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.423451    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj" event={"ID":"a370c44d-12fc-4874-a663-70e1b2fd44ad","Type":"ContainerStarted","Data":"f01590c3c6c53d00d9999c03948f5a51e9acc4bd7f2f83bd455b6804a839ad97"}
Nov 03 20:30:56.426875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.426854    5659 generic.go:356] "Generic (PLEG): container finished" podID="29399fbb-4180-4dcc-b2a0-4a18505e1fb9" containerID="dbac5ea8c3b3fe993b7f57e103fc41cb62765dc8cecf33557b1955b050825ee0" exitCode=60
Nov 03 20:30:56.427044 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.426966    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d" event={"ID":"29399fbb-4180-4dcc-b2a0-4a18505e1fb9","Type":"ContainerDied","Data":"dbac5ea8c3b3fe993b7f57e103fc41cb62765dc8cecf33557b1955b050825ee0"}
Nov 03 20:30:56.427384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.427367    5659 scope.go:122] "RemoveContainer" containerID="dbac5ea8c3b3fe993b7f57e103fc41cb62765dc8cecf33557b1955b050825ee0"
Nov 03 20:30:56.427861 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:56.427832    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d_odh-ai-gateway-infra(29399fbb-4180-4dcc-b2a0-4a18505e1fb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d" podUID="29399fbb-4180-4dcc-b2a0-4a18505e1fb9"
Nov 03 20:30:56.431086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.429694    5659 generic.go:356] "Generic (PLEG): container finished" podID="208a1108-542a-413d-b038-9b087bb636c8" containerID="765ab32725e9bbab5c2eb9c774779cba81707495606f64c838751f45ad26e599" exitCode=60
Nov 03 20:30:56.431086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.429727    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8" event={"ID":"208a1108-542a-413d-b038-9b087bb636c8","Type":"ContainerDied","Data":"765ab32725e9bbab5c2eb9c774779cba81707495606f64c838751f45ad26e599"}
Nov 03 20:30:56.431086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.430047    5659 scope.go:122] "RemoveContainer" containerID="765ab32725e9bbab5c2eb9c774779cba81707495606f64c838751f45ad26e599"
Nov 03 20:30:56.431086 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:56.430415    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8_odh-ai-gateway-infra(208a1108-542a-413d-b038-9b087bb636c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8" podUID="208a1108-542a-413d-b038-9b087bb636c8"
Nov 03 20:30:56.459617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.459535    5659 generic.go:356] "Generic (PLEG): container finished" podID="31961687-b15e-45a0-856a-874eb4baa3ec" containerID="202503bb3be86f37ab89740b266683337c768be32e79686b302e0bb8d2a20400" exitCode=60
Nov 03 20:30:56.460785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.459987    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5" event={"ID":"31961687-b15e-45a0-856a-874eb4baa3ec","Type":"ContainerDied","Data":"202503bb3be86f37ab89740b266683337c768be32e79686b302e0bb8d2a20400"}
Nov 03 20:30:56.460785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.460591    5659 scope.go:122] "RemoveContainer" containerID="202503bb3be86f37ab89740b266683337c768be32e79686b302e0bb8d2a20400"
Nov 03 20:30:56.461412 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:56.461354    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5_odh-ai-gateway-infra(31961687-b15e-45a0-856a-874eb4baa3ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5" podUID="31961687-b15e-45a0-856a-874eb4baa3ec"
Nov 03 20:30:56.467263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.467220    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb" event={"ID":"231e523c-84a7-4b9f-9f6c-6b9603566f86","Type":"ContainerStarted","Data":"ed9572e837154b5595bee3fef9bf6b990153749a2d2a4532fea351864c8c87b0"}
Nov 03 20:30:56.482015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.481976    5659 generic.go:356] "Generic (PLEG): container finished" podID="56bf2d77-4688-4756-a9bb-01f7b346457c" containerID="07ac5c968ed7900c1dc38b78c7f3ee5783c27e8e0b752dbba411fcfa4e85e08f" exitCode=22
Nov 03 20:30:56.482276 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.482244    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj" event={"ID":"56bf2d77-4688-4756-a9bb-01f7b346457c","Type":"ContainerDied","Data":"07ac5c968ed7900c1dc38b78c7f3ee5783c27e8e0b752dbba411fcfa4e85e08f"}
Nov 03 20:30:56.482955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.482935    5659 scope.go:122] "RemoveContainer" containerID="07ac5c968ed7900c1dc38b78c7f3ee5783c27e8e0b752dbba411fcfa4e85e08f"
Nov 03 20:30:56.483522 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:56.483467    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-tbfnj_odh-ai-gateway-infra(56bf2d77-4688-4756-a9bb-01f7b346457c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj" podUID="56bf2d77-4688-4756-a9bb-01f7b346457c"
Nov 03 20:30:56.495394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.495369    5659 scope.go:122] "RemoveContainer" containerID="10552455d97167d331ad974dfc9f03db46c2f580a755803f5fde776f9550e6b5"
Nov 03 20:30:56.498422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.498399    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh" event={"ID":"bcc5e85b-47e2-44f9-990c-7cd7780e3256","Type":"ContainerStarted","Data":"f68f204c42c7810cc329ad56898722830ce3e8090b2d934d7dc5d831bea5ef3e"}
Nov 03 20:30:56.522082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.522040    5659 scope.go:122] "RemoveContainer" containerID="57d09492dc416b3953e3cc607fbd35003563df74d1cf21fe0762b5bf6e2316da"
Nov 03 20:30:56.629969 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.629939    5659 scope.go:122] "RemoveContainer" containerID="abfbef523a1ff7b63c1b5227496e7bae8babc2bca5451bab96dbfc3ade2c97c4"
Nov 03 20:30:56.757920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.755905    5659 scope.go:122] "RemoveContainer" containerID="34910a865d025d602b2a18da8434e16353333987801bbc020cefb378bf6f54f7"
Nov 03 20:30:56.828409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.828086    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"
Nov 03 20:30:56.830715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.830674    5659 scope.go:122] "RemoveContainer" containerID="e51baa26be341530efd3541b8d76a60d5d44383684b31e7bc17a2c9a6d6ef3dd"
Nov 03 20:30:56.831708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.831678    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15e208a-f352-452c-b139-9526f846d5ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15e208a-f352-452c-b139-9526f846d5ab-maas-api-service-ca\") pod \"a15e208a-f352-452c-b139-9526f846d5ab\" (UID: \"a15e208a-f352-452c-b139-9526f846d5ab\") "
Nov 03 20:30:56.832899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.832874    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15e208a-f352-452c-b139-9526f846d5ab-maas-api-service-ca" pod "a15e208a-f352-452c-b139-9526f846d5ab" (UID: "a15e208a-f352-452c-b139-9526f846d5ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:56.912340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.865684    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15e208a-f352-452c-b139-9526f846d5ab-kube-api-access-8s22s\" (UniqueName: \"kubernetes.io/projected/a15e208a-f352-452c-b139-9526f846d5ab-kube-api-access-8s22s\") pod \"a15e208a-f352-452c-b139-9526f846d5ab\" (UID: \"a15e208a-f352-452c-b139-9526f846d5ab\") "
Nov 03 20:30:56.912340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.866741    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15e208a-f352-452c-b139-9526f846d5ab-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:56.912340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.884231    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"]
Nov 03 20:30:56.912340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.885703    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a15e208a-f352-452c-b139-9526f846d5ab" containerName="cleanup"
Nov 03 20:30:56.912340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.885724    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15e208a-f352-452c-b139-9526f846d5ab" containerName="cleanup"
Nov 03 20:30:56.912340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.885843    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a15e208a-f352-452c-b139-9526f846d5ab" containerName="cleanup"
Nov 03 20:30:56.912340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.885853    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15e208a-f352-452c-b139-9526f846d5ab" containerName="cleanup"
Nov 03 20:30:56.912340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.886081    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15e208a-f352-452c-b139-9526f846d5ab" containerName="cleanup"
Nov 03 20:30:56.912340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.886102    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15e208a-f352-452c-b139-9526f846d5ab" containerName="cleanup"
Nov 03 20:30:56.912340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.886117    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15e208a-f352-452c-b139-9526f846d5ab" containerName="cleanup"
Nov 03 20:30:56.912340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.900472    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"]
Nov 03 20:30:56.912340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.900615    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"
Nov 03 20:30:56.915278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:56.915183    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15e208a-f352-452c-b139-9526f846d5ab-kube-api-access-8s22s" pod "a15e208a-f352-452c-b139-9526f846d5ab" (UID: "a15e208a-f352-452c-b139-9526f846d5ab"). InnerVolumeSpecName "kube-api-access-8s22s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:57.007427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.007393    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blqgv\" (UniqueName: \"kubernetes.io/projected/9e43233f-4023-428a-8cff-fffd9ab19d74-kube-api-access-blqgv\") pod \"maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw\" (UID: \"9e43233f-4023-428a-8cff-fffd9ab19d74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"
Nov 03 20:30:57.007849 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.007826    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e43233f-4023-428a-8cff-fffd9ab19d74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw\" (UID: \"9e43233f-4023-428a-8cff-fffd9ab19d74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"
Nov 03 20:30:57.008073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.008055    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s22s\" (UniqueName: \"kubernetes.io/projected/a15e208a-f352-452c-b139-9526f846d5ab-kube-api-access-8s22s\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:57.110449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.110317    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e43233f-4023-428a-8cff-fffd9ab19d74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw\" (UID: \"9e43233f-4023-428a-8cff-fffd9ab19d74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"
Nov 03 20:30:57.110682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.110503    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blqgv\" (UniqueName: \"kubernetes.io/projected/9e43233f-4023-428a-8cff-fffd9ab19d74-kube-api-access-blqgv\") pod \"maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw\" (UID: \"9e43233f-4023-428a-8cff-fffd9ab19d74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"
Nov 03 20:30:57.111166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.111128    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e43233f-4023-428a-8cff-fffd9ab19d74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw\" (UID: \"9e43233f-4023-428a-8cff-fffd9ab19d74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"
Nov 03 20:30:57.123381 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.123355    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blqgv\" (UniqueName: \"kubernetes.io/projected/9e43233f-4023-428a-8cff-fffd9ab19d74-kube-api-access-blqgv\") pod \"maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw\" (UID: \"9e43233f-4023-428a-8cff-fffd9ab19d74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"
Nov 03 20:30:57.227110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.227076    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"
Nov 03 20:30:57.347958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.345067    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:57.725041 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.725005    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"]
Nov 03 20:30:57.727499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.727435    5659 generic.go:356] "Generic (PLEG): container finished" podID="a370c44d-12fc-4874-a663-70e1b2fd44ad" containerID="f01590c3c6c53d00d9999c03948f5a51e9acc4bd7f2f83bd455b6804a839ad97" exitCode=60
Nov 03 20:30:57.728228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.728205    5659 scope.go:122] "RemoveContainer" containerID="f01590c3c6c53d00d9999c03948f5a51e9acc4bd7f2f83bd455b6804a839ad97"
Nov 03 20:30:57.730940 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.730894    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj" event={"ID":"a370c44d-12fc-4874-a663-70e1b2fd44ad","Type":"ContainerDied","Data":"f01590c3c6c53d00d9999c03948f5a51e9acc4bd7f2f83bd455b6804a839ad97"}
Nov 03 20:30:57.731185 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.731168    5659 scope.go:122] "RemoveContainer" containerID="b579c803c5fe631d9c0c211f62f053000cb13fd93509dfe1b638826297c62db4"
Nov 03 20:30:57.756506 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:57.735759    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj_odh-ai-gateway-infra(a370c44d-12fc-4874-a663-70e1b2fd44ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj" podUID="a370c44d-12fc-4874-a663-70e1b2fd44ad"
Nov 03 20:30:57.805978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.805939    5659 generic.go:356] "Generic (PLEG): container finished" podID="231e523c-84a7-4b9f-9f6c-6b9603566f86" containerID="ed9572e837154b5595bee3fef9bf6b990153749a2d2a4532fea351864c8c87b0" exitCode=60
Nov 03 20:30:57.806249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.806221    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb" event={"ID":"231e523c-84a7-4b9f-9f6c-6b9603566f86","Type":"ContainerDied","Data":"ed9572e837154b5595bee3fef9bf6b990153749a2d2a4532fea351864c8c87b0"}
Nov 03 20:30:57.823505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.823142    5659 scope.go:122] "RemoveContainer" containerID="765ab32725e9bbab5c2eb9c774779cba81707495606f64c838751f45ad26e599"
Nov 03 20:30:57.824130 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:57.823664    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8_odh-ai-gateway-infra(208a1108-542a-413d-b038-9b087bb636c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8" podUID="208a1108-542a-413d-b038-9b087bb636c8"
Nov 03 20:30:57.824130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.823902    5659 scope.go:122] "RemoveContainer" containerID="202503bb3be86f37ab89740b266683337c768be32e79686b302e0bb8d2a20400"
Nov 03 20:30:57.824321 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:57.824205    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5_odh-ai-gateway-infra(31961687-b15e-45a0-856a-874eb4baa3ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5" podUID="31961687-b15e-45a0-856a-874eb4baa3ec"
Nov 03 20:30:57.824660 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.824638    5659 scope.go:122] "RemoveContainer" containerID="884d016e21ff67955fb3b9ea6fe77f1d56ab897fdf56a36fde75ea3fd72a60ff"
Nov 03 20:30:57.827509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.827166    5659 scope.go:122] "RemoveContainer" containerID="07ac5c968ed7900c1dc38b78c7f3ee5783c27e8e0b752dbba411fcfa4e85e08f"
Nov 03 20:30:57.827815 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:57.827534    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-tbfnj_odh-ai-gateway-infra(56bf2d77-4688-4756-a9bb-01f7b346457c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj" podUID="56bf2d77-4688-4756-a9bb-01f7b346457c"
Nov 03 20:30:57.830700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.830678    5659 generic.go:356] "Generic (PLEG): container finished" podID="bcc5e85b-47e2-44f9-990c-7cd7780e3256" containerID="f68f204c42c7810cc329ad56898722830ce3e8090b2d934d7dc5d831bea5ef3e" exitCode=60
Nov 03 20:30:57.830832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.830806    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh" event={"ID":"bcc5e85b-47e2-44f9-990c-7cd7780e3256","Type":"ContainerDied","Data":"f68f204c42c7810cc329ad56898722830ce3e8090b2d934d7dc5d831bea5ef3e"}
Nov 03 20:30:57.831054 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.831034    5659 scope.go:122] "RemoveContainer" containerID="f68f204c42c7810cc329ad56898722830ce3e8090b2d934d7dc5d831bea5ef3e"
Nov 03 20:30:57.831353 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:57.831328    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-kn9fh_odh-ai-gateway-infra(bcc5e85b-47e2-44f9-990c-7cd7780e3256)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh" podUID="bcc5e85b-47e2-44f9-990c-7cd7780e3256"
Nov 03 20:30:57.834565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.833827    5659 generic.go:356] "Generic (PLEG): container finished" podID="f938b051-c9cc-4600-89b3-f2a4fde514ec" containerID="ef8dcbbb40cad41f8fd0f1ea5429ccdd1ad574ac875260905b06c62b68ca2790" exitCode=60
Nov 03 20:30:57.834565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.833937    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc" event={"ID":"f938b051-c9cc-4600-89b3-f2a4fde514ec","Type":"ContainerDied","Data":"ef8dcbbb40cad41f8fd0f1ea5429ccdd1ad574ac875260905b06c62b68ca2790"}
Nov 03 20:30:57.834565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.834327    5659 scope.go:122] "RemoveContainer" containerID="ef8dcbbb40cad41f8fd0f1ea5429ccdd1ad574ac875260905b06c62b68ca2790"
Nov 03 20:30:57.835030 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:57.834973    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc_odh-ai-gateway-infra(f938b051-c9cc-4600-89b3-f2a4fde514ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc" podUID="f938b051-c9cc-4600-89b3-f2a4fde514ec"
Nov 03 20:30:57.838124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.837781    5659 scope.go:122] "RemoveContainer" containerID="e83f41bcae8b8894f2c9e53468d1c5eb8a6ffca9108f56551d22e419c0c0a3cc"
Nov 03 20:30:57.838124 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:57.838075    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc_odh-ai-gateway-infra(2bc004c1-979c-4f3e-ba19-fddcd5e58629)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc" podUID="2bc004c1-979c-4f3e-ba19-fddcd5e58629"
Nov 03 20:30:57.841441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.841408    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd" event={"ID":"d6e2428c-648d-4ef4-bed2-8b50adb40fe2","Type":"ContainerStarted","Data":"c05d5a27869f5112a39c6a0a93ad1be5b32f6ac79d5be3c2d59e027a445776c2"}
Nov 03 20:30:57.842061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.842045    5659 scope.go:122] "RemoveContainer" containerID="c05d5a27869f5112a39c6a0a93ad1be5b32f6ac79d5be3c2d59e027a445776c2"
Nov 03 20:30:57.842589 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:57.842552    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-tfnfd_odh-ai-gateway-infra(d6e2428c-648d-4ef4-bed2-8b50adb40fe2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2"
Nov 03 20:30:57.845505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.845473    5659 scope.go:122] "RemoveContainer" containerID="1d12f1d96bca9eb3f7e90b44cd41a47d2e52331b1dadbd7aa26ed05a077a9742"
Nov 03 20:30:57.845912 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:57.845885    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v_odh-ai-gateway-infra(c1c13e2c-aeb0-4eb2-90ed-78c91e40280e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v" podUID="c1c13e2c-aeb0-4eb2-90ed-78c91e40280e"
Nov 03 20:30:57.851132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.851111    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"
Nov 03 20:30:57.855679 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.851281    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z" event={"ID":"a15e208a-f352-452c-b139-9526f846d5ab","Type":"ContainerDied","Data":"5030333b675c5fa6bb317804a3f5cec8626f20487ec27bb1df80caa660e19866"}
Nov 03 20:30:57.857375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.857350    5659 generic.go:356] "Generic (PLEG): container finished" podID="f18783ab-02f5-4154-bc33-bc738babbd40" containerID="dad3ee195739fad2412a2832a8ae3ca6a545960f3bbf7ef8025dafee17444b38" exitCode=60
Nov 03 20:30:57.857566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.857547    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks" event={"ID":"f18783ab-02f5-4154-bc33-bc738babbd40","Type":"ContainerDied","Data":"dad3ee195739fad2412a2832a8ae3ca6a545960f3bbf7ef8025dafee17444b38"}
Nov 03 20:30:57.871628 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:57.871601    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e43233f_4023_428a_8cff_fffd9ab19d74.slice/crio-5ebc0bf36ed010a6294575570fc70d49ddbb8779a9f1c05a444b6e902990a0d0 WatchSource:0}: Error finding container 5ebc0bf36ed010a6294575570fc70d49ddbb8779a9f1c05a444b6e902990a0d0: Status 404 returned error can't find the container with id 5ebc0bf36ed010a6294575570fc70d49ddbb8779a9f1c05a444b6e902990a0d0
Nov 03 20:30:57.873560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.873469    5659 scope.go:122] "RemoveContainer" containerID="dad3ee195739fad2412a2832a8ae3ca6a545960f3bbf7ef8025dafee17444b38"
Nov 03 20:30:57.874042 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:57.873999    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks_odh-ai-gateway-infra(f18783ab-02f5-4154-bc33-bc738babbd40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks" podUID="f18783ab-02f5-4154-bc33-bc738babbd40"
Nov 03 20:30:57.916706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.916249    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"]
Nov 03 20:30:57.926981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.926954    5659 scope.go:122] "RemoveContainer" containerID="44bb1e8713cf1cee7b57284e64d367cec45fa1e4d12d71e180f7fd1bb9fafe1d"
Nov 03 20:30:57.965182 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:57.965148    5659 scope.go:122] "RemoveContainer" containerID="ff7d07e674fd3e622c2d85ebb06069df5aae590dec26df3439672ea210b2b59f"
Nov 03 20:30:58.036578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.036455    5659 scope.go:122] "RemoveContainer" containerID="2e89c6c14cf0a704a857fecd1d736f111fe590801b74f81adb9e51e75d467eb2"
Nov 03 20:30:58.090370 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.090339    5659 scope.go:122] "RemoveContainer" containerID="40f69d0dfac546b848125fe0f46ee0a91147cf9e63ee7a69de362282ea4ce63e"
Nov 03 20:30:58.100873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.100843    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"]
Nov 03 20:30:58.115862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.115835    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895630-wdv2z"]
Nov 03 20:30:58.134031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.134003    5659 scope.go:122] "RemoveContainer" containerID="35c900483b4a6d3fa8b223d646ff00ad132d2f00e17d2176bab0fe139c80ea35"
Nov 03 20:30:58.292376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.292352    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"
Nov 03 20:30:58.326603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.326572    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"]
Nov 03 20:30:58.327578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.327561    5659 cpu_manager.go:414] "containerMap: removing container" podUID="231e523c-84a7-4b9f-9f6c-6b9603566f86" containerName="cleanup"
Nov 03 20:30:58.327632 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.327578    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231e523c-84a7-4b9f-9f6c-6b9603566f86" containerName="cleanup"
Nov 03 20:30:58.327632 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.327602    5659 cpu_manager.go:414] "containerMap: removing container" podUID="231e523c-84a7-4b9f-9f6c-6b9603566f86" containerName="cleanup"
Nov 03 20:30:58.327632 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.327608    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231e523c-84a7-4b9f-9f6c-6b9603566f86" containerName="cleanup"
Nov 03 20:30:58.327731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.327657    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a15e208a-f352-452c-b139-9526f846d5ab" containerName="cleanup"
Nov 03 20:30:58.327731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.327663    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15e208a-f352-452c-b139-9526f846d5ab" containerName="cleanup"
Nov 03 20:30:58.327796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.327792    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="231e523c-84a7-4b9f-9f6c-6b9603566f86" containerName="cleanup"
Nov 03 20:30:58.327828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.327803    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="231e523c-84a7-4b9f-9f6c-6b9603566f86" containerName="cleanup"
Nov 03 20:30:58.327828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.327814    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="231e523c-84a7-4b9f-9f6c-6b9603566f86" containerName="cleanup"
Nov 03 20:30:58.331834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.331808    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"
Nov 03 20:30:58.344204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.344174    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"]
Nov 03 20:30:58.414023 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.413929    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231e523c-84a7-4b9f-9f6c-6b9603566f86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231e523c-84a7-4b9f-9f6c-6b9603566f86-maas-api-service-ca\") pod \"231e523c-84a7-4b9f-9f6c-6b9603566f86\" (UID: \"231e523c-84a7-4b9f-9f6c-6b9603566f86\") "
Nov 03 20:30:58.414471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.414208    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231e523c-84a7-4b9f-9f6c-6b9603566f86-kube-api-access-zk8qh\" (UniqueName: \"kubernetes.io/projected/231e523c-84a7-4b9f-9f6c-6b9603566f86-kube-api-access-zk8qh\") pod \"231e523c-84a7-4b9f-9f6c-6b9603566f86\" (UID: \"231e523c-84a7-4b9f-9f6c-6b9603566f86\") "
Nov 03 20:30:58.414471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.414332    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231e523c-84a7-4b9f-9f6c-6b9603566f86-maas-api-service-ca" pod "231e523c-84a7-4b9f-9f6c-6b9603566f86" (UID: "231e523c-84a7-4b9f-9f6c-6b9603566f86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:58.414944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.414904    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zglvx\" (UniqueName: \"kubernetes.io/projected/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-kube-api-access-zglvx\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99\" (UID: \"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"
Nov 03 20:30:58.415146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.415117    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99\" (UID: \"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"
Nov 03 20:30:58.415620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.415597    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/231e523c-84a7-4b9f-9f6c-6b9603566f86-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:58.417051 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.417022    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231e523c-84a7-4b9f-9f6c-6b9603566f86-kube-api-access-zk8qh" pod "231e523c-84a7-4b9f-9f6c-6b9603566f86" (UID: "231e523c-84a7-4b9f-9f6c-6b9603566f86"). InnerVolumeSpecName "kube-api-access-zk8qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:58.517598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.517552    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zglvx\" (UniqueName: \"kubernetes.io/projected/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-kube-api-access-zglvx\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99\" (UID: \"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"
Nov 03 20:30:58.517817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.517704    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99\" (UID: \"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"
Nov 03 20:30:58.517930 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.517916    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk8qh\" (UniqueName: \"kubernetes.io/projected/231e523c-84a7-4b9f-9f6c-6b9603566f86-kube-api-access-zk8qh\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:58.518359 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.518336    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99\" (UID: \"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"
Nov 03 20:30:58.530205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.530176    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zglvx\" (UniqueName: \"kubernetes.io/projected/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-kube-api-access-zglvx\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99\" (UID: \"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"
Nov 03 20:30:58.666561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.666456    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"
Nov 03 20:30:58.844704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.844642    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:58.975907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.975875    5659 scope.go:122] "RemoveContainer" containerID="f68f204c42c7810cc329ad56898722830ce3e8090b2d934d7dc5d831bea5ef3e"
Nov 03 20:30:58.976326 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:58.976298    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-kn9fh_odh-ai-gateway-infra(bcc5e85b-47e2-44f9-990c-7cd7780e3256)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh" podUID="bcc5e85b-47e2-44f9-990c-7cd7780e3256"
Nov 03 20:30:58.978017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.977995    5659 scope.go:122] "RemoveContainer" containerID="ef8dcbbb40cad41f8fd0f1ea5429ccdd1ad574ac875260905b06c62b68ca2790"
Nov 03 20:30:58.978346 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:58.978326    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc_odh-ai-gateway-infra(f938b051-c9cc-4600-89b3-f2a4fde514ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc" podUID="f938b051-c9cc-4600-89b3-f2a4fde514ec"
Nov 03 20:30:58.980784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.980761    5659 generic.go:356] "Generic (PLEG): container finished" podID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2" containerID="c05d5a27869f5112a39c6a0a93ad1be5b32f6ac79d5be3c2d59e027a445776c2" exitCode=22
Nov 03 20:30:58.980896 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.980840    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd" event={"ID":"d6e2428c-648d-4ef4-bed2-8b50adb40fe2","Type":"ContainerDied","Data":"c05d5a27869f5112a39c6a0a93ad1be5b32f6ac79d5be3c2d59e027a445776c2"}
Nov 03 20:30:58.980951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.980910    5659 scope.go:122] "RemoveContainer" containerID="ef4a0fdfc002af3edda96ca26de19c5cc2e29d2e3acc68f1b0ae803fa533223d"
Nov 03 20:30:58.981251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.981232    5659 scope.go:122] "RemoveContainer" containerID="c05d5a27869f5112a39c6a0a93ad1be5b32f6ac79d5be3c2d59e027a445776c2"
Nov 03 20:30:58.981975 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:58.981666    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-tfnfd_odh-ai-gateway-infra(d6e2428c-648d-4ef4-bed2-8b50adb40fe2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2"
Nov 03 20:30:58.984723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.984693    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw" event={"ID":"9e43233f-4023-428a-8cff-fffd9ab19d74","Type":"ContainerStarted","Data":"48c93a6115111b023f0e77ac5fe136d48cb2ffddd34331eb469886a346d29af8"}
Nov 03 20:30:58.984844 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.984759    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw" event={"ID":"9e43233f-4023-428a-8cff-fffd9ab19d74","Type":"ContainerStarted","Data":"5ebc0bf36ed010a6294575570fc70d49ddbb8779a9f1c05a444b6e902990a0d0"}
Nov 03 20:30:58.986873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.986848    5659 scope.go:122] "RemoveContainer" containerID="dad3ee195739fad2412a2832a8ae3ca6a545960f3bbf7ef8025dafee17444b38"
Nov 03 20:30:58.987207 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:58.987184    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks_odh-ai-gateway-infra(f18783ab-02f5-4154-bc33-bc738babbd40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks" podUID="f18783ab-02f5-4154-bc33-bc738babbd40"
Nov 03 20:30:58.989153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.989129    5659 scope.go:122] "RemoveContainer" containerID="f01590c3c6c53d00d9999c03948f5a51e9acc4bd7f2f83bd455b6804a839ad97"
Nov 03 20:30:58.989520 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:30:58.989455    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj_odh-ai-gateway-infra(a370c44d-12fc-4874-a663-70e1b2fd44ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj" podUID="a370c44d-12fc-4874-a663-70e1b2fd44ad"
Nov 03 20:30:58.990759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.990738    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"]
Nov 03 20:30:58.991263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.991230    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l" event={"ID":"b88035b6-4c8a-4c03-b803-e897a2c86818","Type":"ContainerStarted","Data":"5bf06c691ffc7c029814523c054899ca996a85f5e5f781bd7bc98becc342f263"}
Nov 03 20:30:58.994184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.992923    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb" event={"ID":"231e523c-84a7-4b9f-9f6c-6b9603566f86","Type":"ContainerDied","Data":"90fcfd1e019b1e00373d3ec53d04a370cb72b2af9d98323c429efdc764e022b9"}
Nov 03 20:30:58.994184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:58.992959    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"
Nov 03 20:30:59.004431 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:30:59.003504    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f042faa_25c7_4ce0_9fb8_16fc7633f5d5.slice/crio-ba3b9281b98f4c4a3ba8d291a6d7f4bd0476f472e94be2ddae4c635288ff9113 WatchSource:0}: Error finding container ba3b9281b98f4c4a3ba8d291a6d7f4bd0476f472e94be2ddae4c635288ff9113: Status 404 returned error can't find the container with id ba3b9281b98f4c4a3ba8d291a6d7f4bd0476f472e94be2ddae4c635288ff9113
Nov 03 20:30:59.028469 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:59.028399    5659 scope.go:122] "RemoveContainer" containerID="ed9572e837154b5595bee3fef9bf6b990153749a2d2a4532fea351864c8c87b0"
Nov 03 20:30:59.128432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:59.128393    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"]
Nov 03 20:30:59.133473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:59.133419    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw" podStartSLOduration=59.133402337 podStartE2EDuration="59.133402337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:59.131088363 +0000 UTC m=+3211.285990776" watchObservedRunningTime="2026-11-03 20:30:59.133402337 +0000 UTC m=+3211.288304761"
Nov 03 20:30:59.153459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:59.153424    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"]
Nov 03 20:30:59.160630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:59.160601    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4c2a28-29895630-rlsnb"]
Nov 03 20:30:59.624827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:59.624748    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:59.975544 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:59.975507    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231e523c-84a7-4b9f-9f6c-6b9603566f86" path="/var/lib/kubelet/pods/231e523c-84a7-4b9f-9f6c-6b9603566f86/volumes"
Nov 03 20:30:59.976172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:30:59.976150    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15e208a-f352-452c-b139-9526f846d5ab" path="/var/lib/kubelet/pods/a15e208a-f352-452c-b139-9526f846d5ab/volumes"
Nov 03 20:31:00.003362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.003330    5659 generic.go:356] "Generic (PLEG): container finished" podID="b88035b6-4c8a-4c03-b803-e897a2c86818" containerID="5bf06c691ffc7c029814523c054899ca996a85f5e5f781bd7bc98becc342f263" exitCode=60
Nov 03 20:31:00.003582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.003400    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l" event={"ID":"b88035b6-4c8a-4c03-b803-e897a2c86818","Type":"ContainerDied","Data":"5bf06c691ffc7c029814523c054899ca996a85f5e5f781bd7bc98becc342f263"}
Nov 03 20:31:00.003582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.003460    5659 scope.go:122] "RemoveContainer" containerID="ff7d07e674fd3e622c2d85ebb06069df5aae590dec26df3439672ea210b2b59f"
Nov 03 20:31:00.004074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.004053    5659 scope.go:122] "RemoveContainer" containerID="5bf06c691ffc7c029814523c054899ca996a85f5e5f781bd7bc98becc342f263"
Nov 03 20:31:00.004477 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:00.004445    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l_odh-ai-gateway-infra(b88035b6-4c8a-4c03-b803-e897a2c86818)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l" podUID="b88035b6-4c8a-4c03-b803-e897a2c86818"
Nov 03 20:31:00.008542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.008519    5659 generic.go:356] "Generic (PLEG): container finished" podID="9e43233f-4023-428a-8cff-fffd9ab19d74" containerID="48c93a6115111b023f0e77ac5fe136d48cb2ffddd34331eb469886a346d29af8" exitCode=60
Nov 03 20:31:00.008646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.008620    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw" event={"ID":"9e43233f-4023-428a-8cff-fffd9ab19d74","Type":"ContainerDied","Data":"48c93a6115111b023f0e77ac5fe136d48cb2ffddd34331eb469886a346d29af8"}
Nov 03 20:31:00.008976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.008959    5659 scope.go:122] "RemoveContainer" containerID="48c93a6115111b023f0e77ac5fe136d48cb2ffddd34331eb469886a346d29af8"
Nov 03 20:31:00.014594 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.014574    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99" event={"ID":"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5","Type":"ContainerStarted","Data":"c44b88e61ac20fa7323572a0436e270afee146992d826d8b3836b5523af01330"}
Nov 03 20:31:00.014758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.014733    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99" event={"ID":"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5","Type":"ContainerStarted","Data":"ba3b9281b98f4c4a3ba8d291a6d7f4bd0476f472e94be2ddae4c635288ff9113"}
Nov 03 20:31:00.021435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.021414    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"]
Nov 03 20:31:00.030314 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.030278    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"]
Nov 03 20:31:00.044063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.043422    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"]
Nov 03 20:31:00.054599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.052875    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"]
Nov 03 20:31:00.155251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.155197    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"]
Nov 03 20:31:00.171386 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.169116    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"]
Nov 03 20:31:00.186765 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.186711    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99" podStartSLOduration=60.186692285 podStartE2EDuration="1m0.186692285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:00.092336768 +0000 UTC m=+3212.247239188" watchObservedRunningTime="2026-11-03 20:31:00.186692285 +0000 UTC m=+3212.341594707"
Nov 03 20:31:00.210945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.210912    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"]
Nov 03 20:31:00.219947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.219919    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"]
Nov 03 20:31:00.229753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.228473    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"]
Nov 03 20:31:00.237389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.237326    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"]
Nov 03 20:31:00.264199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.262737    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"]
Nov 03 20:31:00.448586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.435695    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:00.558648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.558616    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"
Nov 03 20:31:00.625134 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.624515    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"]
Nov 03 20:31:00.642157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.642128    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"
Nov 03 20:31:00.689294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.687955    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc5e85b-47e2-44f9-990c-7cd7780e3256-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc5e85b-47e2-44f9-990c-7cd7780e3256-maas-api-service-ca\") pod \"bcc5e85b-47e2-44f9-990c-7cd7780e3256\" (UID: \"bcc5e85b-47e2-44f9-990c-7cd7780e3256\") "
Nov 03 20:31:00.689294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.688088    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc5e85b-47e2-44f9-990c-7cd7780e3256-kube-api-access-fgp45\" (UniqueName: \"kubernetes.io/projected/bcc5e85b-47e2-44f9-990c-7cd7780e3256-kube-api-access-fgp45\") pod \"bcc5e85b-47e2-44f9-990c-7cd7780e3256\" (UID: \"bcc5e85b-47e2-44f9-990c-7cd7780e3256\") "
Nov 03 20:31:00.689294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.688225    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-openshift-ca\") pod \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\" (UID: \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\") "
Nov 03 20:31:00.689294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.688351    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-kube-api-access-rmm8h\" (UniqueName: \"kubernetes.io/projected/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-kube-api-access-rmm8h\") pod \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\" (UID: \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\") "
Nov 03 20:31:00.689294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.688552    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-xks-ca\") pod \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\" (UID: \"d6e2428c-648d-4ef4-bed2-8b50adb40fe2\") "
Nov 03 20:31:00.689294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.688776    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc5e85b-47e2-44f9-990c-7cd7780e3256-maas-api-service-ca" pod "bcc5e85b-47e2-44f9-990c-7cd7780e3256" (UID: "bcc5e85b-47e2-44f9-990c-7cd7780e3256"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:00.689294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.689162    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-openshift-ca" pod "d6e2428c-648d-4ef4-bed2-8b50adb40fe2" (UID: "d6e2428c-648d-4ef4-bed2-8b50adb40fe2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:00.689813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.689372    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc5e85b-47e2-44f9-990c-7cd7780e3256-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:00.689813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.689393    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:00.693575 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.693443    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-xks-ca" pod "d6e2428c-648d-4ef4-bed2-8b50adb40fe2" (UID: "d6e2428c-648d-4ef4-bed2-8b50adb40fe2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:00.699096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.698971    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc5e85b-47e2-44f9-990c-7cd7780e3256-kube-api-access-fgp45" pod "bcc5e85b-47e2-44f9-990c-7cd7780e3256" (UID: "bcc5e85b-47e2-44f9-990c-7cd7780e3256"). InnerVolumeSpecName "kube-api-access-fgp45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:00.703610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.702917    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-kube-api-access-rmm8h" pod "d6e2428c-648d-4ef4-bed2-8b50adb40fe2" (UID: "d6e2428c-648d-4ef4-bed2-8b50adb40fe2"). InnerVolumeSpecName "kube-api-access-rmm8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:00.750892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.750528    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"]
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752034    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bcc5e85b-47e2-44f9-990c-7cd7780e3256" containerName="cleanup"
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752057    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc5e85b-47e2-44f9-990c-7cd7780e3256" containerName="cleanup"
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752069    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2" containerName="revoke-keys"
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752074    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2" containerName="revoke-keys"
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752089    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2" containerName="revoke-keys"
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752094    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2" containerName="revoke-keys"
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752122    5659 cpu_manager.go:414] "containerMap: removing container" podUID="231e523c-84a7-4b9f-9f6c-6b9603566f86" containerName="cleanup"
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752127    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231e523c-84a7-4b9f-9f6c-6b9603566f86" containerName="cleanup"
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752136    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2" containerName="revoke-keys"
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752140    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2" containerName="revoke-keys"
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752289    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc5e85b-47e2-44f9-990c-7cd7780e3256" containerName="cleanup"
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752298    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2" containerName="revoke-keys"
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752307    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2" containerName="revoke-keys"
Nov 03 20:31:00.753380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.752316    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc5e85b-47e2-44f9-990c-7cd7780e3256" containerName="cleanup"
Nov 03 20:31:00.758954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.758930    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"
Nov 03 20:31:00.786742 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.786716    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"]
Nov 03 20:31:00.788981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.788568    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"
Nov 03 20:31:00.806410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.806364    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-knsvw\" (UID: \"1c1247bb-a71b-4f74-8261-c56a31e13462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"
Nov 03 20:31:00.807140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.807113    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-knsvw\" (UID: \"1c1247bb-a71b-4f74-8261-c56a31e13462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"
Nov 03 20:31:00.807997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.807975    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b74w\" (UniqueName: \"kubernetes.io/projected/1c1247bb-a71b-4f74-8261-c56a31e13462-kube-api-access-4b74w\") pod \"maas-api-revoke-sub-fb827799-knsvw\" (UID: \"1c1247bb-a71b-4f74-8261-c56a31e13462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"
Nov 03 20:31:00.808582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.808492    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmm8h\" (UniqueName: \"kubernetes.io/projected/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-kube-api-access-rmm8h\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:00.808735 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.808718    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e2428c-648d-4ef4-bed2-8b50adb40fe2-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:00.808832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.808820    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgp45\" (UniqueName: \"kubernetes.io/projected/bcc5e85b-47e2-44f9-990c-7cd7780e3256-kube-api-access-fgp45\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:00.816708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.816685    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"]
Nov 03 20:31:00.833058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.833015    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f938b051-c9cc-4600-89b3-f2a4fde514ec" containerName="cleanup"
Nov 03 20:31:00.833277 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.833252    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f938b051-c9cc-4600-89b3-f2a4fde514ec" containerName="cleanup"
Nov 03 20:31:00.833408 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.833396    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f938b051-c9cc-4600-89b3-f2a4fde514ec" containerName="cleanup"
Nov 03 20:31:00.833511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.833476    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f938b051-c9cc-4600-89b3-f2a4fde514ec" containerName="cleanup"
Nov 03 20:31:00.833672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.833646    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bcc5e85b-47e2-44f9-990c-7cd7780e3256" containerName="cleanup"
Nov 03 20:31:00.833776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.833763    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc5e85b-47e2-44f9-990c-7cd7780e3256" containerName="cleanup"
Nov 03 20:31:00.834177 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.834149    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f938b051-c9cc-4600-89b3-f2a4fde514ec" containerName="cleanup"
Nov 03 20:31:00.834312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.834298    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2" containerName="revoke-keys"
Nov 03 20:31:00.839587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.839557    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"]
Nov 03 20:31:00.839726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.839690    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"
Nov 03 20:31:00.925953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.925920    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f938b051-c9cc-4600-89b3-f2a4fde514ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f938b051-c9cc-4600-89b3-f2a4fde514ec-maas-api-service-ca\") pod \"f938b051-c9cc-4600-89b3-f2a4fde514ec\" (UID: \"f938b051-c9cc-4600-89b3-f2a4fde514ec\") "
Nov 03 20:31:00.927944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.927873    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f938b051-c9cc-4600-89b3-f2a4fde514ec-kube-api-access-8gtfc\" (UniqueName: \"kubernetes.io/projected/f938b051-c9cc-4600-89b3-f2a4fde514ec-kube-api-access-8gtfc\") pod \"f938b051-c9cc-4600-89b3-f2a4fde514ec\" (UID: \"f938b051-c9cc-4600-89b3-f2a4fde514ec\") "
Nov 03 20:31:00.929204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.929175    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b74w\" (UniqueName: \"kubernetes.io/projected/1c1247bb-a71b-4f74-8261-c56a31e13462-kube-api-access-4b74w\") pod \"maas-api-revoke-sub-fb827799-knsvw\" (UID: \"1c1247bb-a71b-4f74-8261-c56a31e13462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"
Nov 03 20:31:00.939087 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.937755    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f938b051-c9cc-4600-89b3-f2a4fde514ec-maas-api-service-ca" pod "f938b051-c9cc-4600-89b3-f2a4fde514ec" (UID: "f938b051-c9cc-4600-89b3-f2a4fde514ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:00.941876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.941846    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-fmd6b\" (UID: \"0005549b-99f3-4bd9-b80d-3414d16e1389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"
Nov 03 20:31:00.946464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.946387    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-knsvw\" (UID: \"1c1247bb-a71b-4f74-8261-c56a31e13462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"
Nov 03 20:31:00.946898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.946868    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f938b051-c9cc-4600-89b3-f2a4fde514ec-kube-api-access-8gtfc" pod "f938b051-c9cc-4600-89b3-f2a4fde514ec" (UID: "f938b051-c9cc-4600-89b3-f2a4fde514ec"). InnerVolumeSpecName "kube-api-access-8gtfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:00.949732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.948810    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-knsvw\" (UID: \"1c1247bb-a71b-4f74-8261-c56a31e13462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"
Nov 03 20:31:00.951993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.951968    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls4bd\" (UniqueName: \"kubernetes.io/projected/0005549b-99f3-4bd9-b80d-3414d16e1389-kube-api-access-ls4bd\") pod \"maas-api-revoke-sub-abad4031-fmd6b\" (UID: \"0005549b-99f3-4bd9-b80d-3414d16e1389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"
Nov 03 20:31:00.952582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.952563    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-fmd6b\" (UID: \"0005549b-99f3-4bd9-b80d-3414d16e1389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"
Nov 03 20:31:00.952835 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.952822    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-knsvw\" (UID: \"1c1247bb-a71b-4f74-8261-c56a31e13462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"
Nov 03 20:31:00.953041 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.953031    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f938b051-c9cc-4600-89b3-f2a4fde514ec-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:00.954439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.954426    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gtfc\" (UniqueName: \"kubernetes.io/projected/f938b051-c9cc-4600-89b3-f2a4fde514ec-kube-api-access-8gtfc\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:00.954534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.952893    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"
Nov 03 20:31:00.955430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.955000    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"
Nov 03 20:31:00.957713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.957670    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b74w\" (UniqueName: \"kubernetes.io/projected/1c1247bb-a71b-4f74-8261-c56a31e13462-kube-api-access-4b74w\") pod \"maas-api-revoke-sub-fb827799-knsvw\" (UID: \"1c1247bb-a71b-4f74-8261-c56a31e13462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"
Nov 03 20:31:00.958719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.958158    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-knsvw\" (UID: \"1c1247bb-a71b-4f74-8261-c56a31e13462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"
Nov 03 20:31:00.965675 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.965370    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"
Nov 03 20:31:00.972058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.972040    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"
Nov 03 20:31:00.996079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.996046    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"
Nov 03 20:31:00.998394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:00.998363    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"
Nov 03 20:31:01.031396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.031319    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"
Nov 03 20:31:01.034356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.034317    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd" event={"ID":"d6e2428c-648d-4ef4-bed2-8b50adb40fe2","Type":"ContainerDied","Data":"402ed13fbb07dab640a3390d640af84c12dd5910fa183a0b0ea40e86ff9e9db8"}
Nov 03 20:31:01.034619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.034598    5659 scope.go:122] "RemoveContainer" containerID="c05d5a27869f5112a39c6a0a93ad1be5b32f6ac79d5be3c2d59e027a445776c2"
Nov 03 20:31:01.046893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.040415    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v" event={"ID":"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e","Type":"ContainerDied","Data":"a35d564c16229896b830f804bcdaae4a510ada1caed29fdfd8dbc22854fcd53b"}
Nov 03 20:31:01.046893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.040614    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"
Nov 03 20:31:01.046893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.043452    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw" event={"ID":"9e43233f-4023-428a-8cff-fffd9ab19d74","Type":"ContainerStarted","Data":"9bade9ba2fc56e6228b95c27f46f07c600f7e6687752ed706ed891f8e4670b66"}
Nov 03 20:31:01.046893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.043630    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw" podUID="9e43233f-4023-428a-8cff-fffd9ab19d74" containerName="cleanup" containerID="cri-o://9bade9ba2fc56e6228b95c27f46f07c600f7e6687752ed706ed891f8e4670b66" gracePeriod=30
Nov 03 20:31:01.046893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.046367    5659 generic.go:356] "Generic (PLEG): container finished" podID="8f042faa-25c7-4ce0-9fb8-16fc7633f5d5" containerID="c44b88e61ac20fa7323572a0436e270afee146992d826d8b3836b5523af01330" exitCode=60
Nov 03 20:31:01.046893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.046445    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99" event={"ID":"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5","Type":"ContainerDied","Data":"c44b88e61ac20fa7323572a0436e270afee146992d826d8b3836b5523af01330"}
Nov 03 20:31:01.046893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.046836    5659 scope.go:122] "RemoveContainer" containerID="c44b88e61ac20fa7323572a0436e270afee146992d826d8b3836b5523af01330"
Nov 03 20:31:01.060919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.057414    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208a1108-542a-413d-b038-9b087bb636c8-kube-api-access-8bnsk\" (UniqueName: \"kubernetes.io/projected/208a1108-542a-413d-b038-9b087bb636c8-kube-api-access-8bnsk\") pod \"208a1108-542a-413d-b038-9b087bb636c8\" (UID: \"208a1108-542a-413d-b038-9b087bb636c8\") "
Nov 03 20:31:01.060919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.057466    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208a1108-542a-413d-b038-9b087bb636c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/208a1108-542a-413d-b038-9b087bb636c8-maas-api-service-ca\") pod \"208a1108-542a-413d-b038-9b087bb636c8\" (UID: \"208a1108-542a-413d-b038-9b087bb636c8\") "
Nov 03 20:31:01.060919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.057606    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-maas-api-service-ca\") pod \"29399fbb-4180-4dcc-b2a0-4a18505e1fb9\" (UID: \"29399fbb-4180-4dcc-b2a0-4a18505e1fb9\") "
Nov 03 20:31:01.060919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.057725    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-kube-api-access-2llk9\" (UniqueName: \"kubernetes.io/projected/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-kube-api-access-2llk9\") pod \"29399fbb-4180-4dcc-b2a0-4a18505e1fb9\" (UID: \"29399fbb-4180-4dcc-b2a0-4a18505e1fb9\") "
Nov 03 20:31:01.060919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.058566    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls4bd\" (UniqueName: \"kubernetes.io/projected/0005549b-99f3-4bd9-b80d-3414d16e1389-kube-api-access-ls4bd\") pod \"maas-api-revoke-sub-abad4031-fmd6b\" (UID: \"0005549b-99f3-4bd9-b80d-3414d16e1389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"
Nov 03 20:31:01.060919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.058608    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-fmd6b\" (UID: \"0005549b-99f3-4bd9-b80d-3414d16e1389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"
Nov 03 20:31:01.060919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.058807    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-fmd6b\" (UID: \"0005549b-99f3-4bd9-b80d-3414d16e1389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"
Nov 03 20:31:01.062168 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.061555    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208a1108-542a-413d-b038-9b087bb636c8-maas-api-service-ca" pod "208a1108-542a-413d-b038-9b087bb636c8" (UID: "208a1108-542a-413d-b038-9b087bb636c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:01.062168 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.061898    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-maas-api-service-ca" pod "29399fbb-4180-4dcc-b2a0-4a18505e1fb9" (UID: "29399fbb-4180-4dcc-b2a0-4a18505e1fb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:01.065180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.065159    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj" event={"ID":"a370c44d-12fc-4874-a663-70e1b2fd44ad","Type":"ContainerDied","Data":"799bcc944e9091f0320f596043bf2b2485b1e393fc07539b831d43ee8bce743e"}
Nov 03 20:31:01.065427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.065408    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"
Nov 03 20:31:01.067692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.067673    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d" event={"ID":"29399fbb-4180-4dcc-b2a0-4a18505e1fb9","Type":"ContainerDied","Data":"4e32cfc9f26d151209180016214dc98cf28428a2c765bcd6142f307381d08354"}
Nov 03 20:31:01.067886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.067871    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"
Nov 03 20:31:01.071736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.071590    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8" event={"ID":"208a1108-542a-413d-b038-9b087bb636c8","Type":"ContainerDied","Data":"ac43007cf64c848cebfd3d95d33a8b2ae1ab7c63fe61d768859c8a26d26323e2"}
Nov 03 20:31:01.073675 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.073578    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"
Nov 03 20:31:01.083390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.083367    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-fmd6b\" (UID: \"0005549b-99f3-4bd9-b80d-3414d16e1389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"
Nov 03 20:31:01.089609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.089584    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"
Nov 03 20:31:01.089723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.089668    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5" event={"ID":"31961687-b15e-45a0-856a-874eb4baa3ec","Type":"ContainerDied","Data":"e4098e3e66049b612b54000faaf03366b5d48f250cef3266a6ca3910f3ac4257"}
Nov 03 20:31:01.100661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.097444    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-kube-api-access-2llk9" pod "29399fbb-4180-4dcc-b2a0-4a18505e1fb9" (UID: "29399fbb-4180-4dcc-b2a0-4a18505e1fb9"). InnerVolumeSpecName "kube-api-access-2llk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:01.100661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.097696    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208a1108-542a-413d-b038-9b087bb636c8-kube-api-access-8bnsk" pod "208a1108-542a-413d-b038-9b087bb636c8" (UID: "208a1108-542a-413d-b038-9b087bb636c8"). InnerVolumeSpecName "kube-api-access-8bnsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:01.100661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.099477    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls4bd\" (UniqueName: \"kubernetes.io/projected/0005549b-99f3-4bd9-b80d-3414d16e1389-kube-api-access-ls4bd\") pod \"maas-api-revoke-sub-abad4031-fmd6b\" (UID: \"0005549b-99f3-4bd9-b80d-3414d16e1389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"
Nov 03 20:31:01.100661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.100209    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-fmd6b\" (UID: \"0005549b-99f3-4bd9-b80d-3414d16e1389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"
Nov 03 20:31:01.107156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.107129    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"
Nov 03 20:31:01.108835 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.108815    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh" event={"ID":"bcc5e85b-47e2-44f9-990c-7cd7780e3256","Type":"ContainerDied","Data":"a96fbe5a17eb9d4fe19a08ca9884463832d4daacd6b3968fc1338d78a2bef2d6"}
Nov 03 20:31:01.109029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.109016    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"
Nov 03 20:31:01.131850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.131539    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"
Nov 03 20:31:01.132878 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.131607    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc" event={"ID":"f938b051-c9cc-4600-89b3-f2a4fde514ec","Type":"ContainerDied","Data":"5caf6e37eb0d159f2fd2125ea473e4937790bacca73bb7a8d7c07f72ebc1b463"}
Nov 03 20:31:01.161149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.159719    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-maas-api-service-ca\") pod \"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e\" (UID: \"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e\") "
Nov 03 20:31:01.161149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.159806    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31961687-b15e-45a0-856a-874eb4baa3ec-kube-api-access-p6t4w\" (UniqueName: \"kubernetes.io/projected/31961687-b15e-45a0-856a-874eb4baa3ec-kube-api-access-p6t4w\") pod \"31961687-b15e-45a0-856a-874eb4baa3ec\" (UID: \"31961687-b15e-45a0-856a-874eb4baa3ec\") "
Nov 03 20:31:01.161149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.160015    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc004c1-979c-4f3e-ba19-fddcd5e58629-kube-api-access-r59h9\" (UniqueName: \"kubernetes.io/projected/2bc004c1-979c-4f3e-ba19-fddcd5e58629-kube-api-access-r59h9\") pod \"2bc004c1-979c-4f3e-ba19-fddcd5e58629\" (UID: \"2bc004c1-979c-4f3e-ba19-fddcd5e58629\") "
Nov 03 20:31:01.161149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.160053    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc004c1-979c-4f3e-ba19-fddcd5e58629-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc004c1-979c-4f3e-ba19-fddcd5e58629-maas-api-service-ca\") pod \"2bc004c1-979c-4f3e-ba19-fddcd5e58629\" (UID: \"2bc004c1-979c-4f3e-ba19-fddcd5e58629\") "
Nov 03 20:31:01.161149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.160161    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-kube-api-access-jtrl6\" (UniqueName: \"kubernetes.io/projected/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-kube-api-access-jtrl6\") pod \"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e\" (UID: \"c1c13e2c-aeb0-4eb2-90ed-78c91e40280e\") "
Nov 03 20:31:01.161149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.160195    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a370c44d-12fc-4874-a663-70e1b2fd44ad-kube-api-access-b2xz9\" (UniqueName: \"kubernetes.io/projected/a370c44d-12fc-4874-a663-70e1b2fd44ad-kube-api-access-b2xz9\") pod \"a370c44d-12fc-4874-a663-70e1b2fd44ad\" (UID: \"a370c44d-12fc-4874-a663-70e1b2fd44ad\") "
Nov 03 20:31:01.161149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.160228    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31961687-b15e-45a0-856a-874eb4baa3ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31961687-b15e-45a0-856a-874eb4baa3ec-maas-api-service-ca\") pod \"31961687-b15e-45a0-856a-874eb4baa3ec\" (UID: \"31961687-b15e-45a0-856a-874eb4baa3ec\") "
Nov 03 20:31:01.161149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.160282    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a370c44d-12fc-4874-a663-70e1b2fd44ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a370c44d-12fc-4874-a663-70e1b2fd44ad-maas-api-service-ca\") pod \"a370c44d-12fc-4874-a663-70e1b2fd44ad\" (UID: \"a370c44d-12fc-4874-a663-70e1b2fd44ad\") "
Nov 03 20:31:01.161149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.160862    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-maas-api-service-ca" pod "c1c13e2c-aeb0-4eb2-90ed-78c91e40280e" (UID: "c1c13e2c-aeb0-4eb2-90ed-78c91e40280e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:01.161753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.161609    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bnsk\" (UniqueName: \"kubernetes.io/projected/208a1108-542a-413d-b038-9b087bb636c8-kube-api-access-8bnsk\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.161753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.161636    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/208a1108-542a-413d-b038-9b087bb636c8-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.161753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.161652    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.161753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.161668    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2llk9\" (UniqueName: \"kubernetes.io/projected/29399fbb-4180-4dcc-b2a0-4a18505e1fb9-kube-api-access-2llk9\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.161753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.161684    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.161753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.161694    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a370c44d-12fc-4874-a663-70e1b2fd44ad-maas-api-service-ca" pod "a370c44d-12fc-4874-a663-70e1b2fd44ad" (UID: "a370c44d-12fc-4874-a663-70e1b2fd44ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:01.162075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.162034    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc004c1-979c-4f3e-ba19-fddcd5e58629-maas-api-service-ca" pod "2bc004c1-979c-4f3e-ba19-fddcd5e58629" (UID: "2bc004c1-979c-4f3e-ba19-fddcd5e58629"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:01.162914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.162879    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31961687-b15e-45a0-856a-874eb4baa3ec-maas-api-service-ca" pod "31961687-b15e-45a0-856a-874eb4baa3ec" (UID: "31961687-b15e-45a0-856a-874eb4baa3ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:01.165678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.165638    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31961687-b15e-45a0-856a-874eb4baa3ec-kube-api-access-p6t4w" pod "31961687-b15e-45a0-856a-874eb4baa3ec" (UID: "31961687-b15e-45a0-856a-874eb4baa3ec"). InnerVolumeSpecName "kube-api-access-p6t4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:01.167116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.167054    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-kube-api-access-jtrl6" pod "c1c13e2c-aeb0-4eb2-90ed-78c91e40280e" (UID: "c1c13e2c-aeb0-4eb2-90ed-78c91e40280e"). InnerVolumeSpecName "kube-api-access-jtrl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:01.167116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.167096    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a370c44d-12fc-4874-a663-70e1b2fd44ad-kube-api-access-b2xz9" pod "a370c44d-12fc-4874-a663-70e1b2fd44ad" (UID: "a370c44d-12fc-4874-a663-70e1b2fd44ad"). InnerVolumeSpecName "kube-api-access-b2xz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:01.167748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.167559    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc004c1-979c-4f3e-ba19-fddcd5e58629-kube-api-access-r59h9" pod "2bc004c1-979c-4f3e-ba19-fddcd5e58629" (UID: "2bc004c1-979c-4f3e-ba19-fddcd5e58629"). InnerVolumeSpecName "kube-api-access-r59h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:01.171626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.169914    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc" event={"ID":"2bc004c1-979c-4f3e-ba19-fddcd5e58629","Type":"ContainerDied","Data":"7652a14b23cd7db884deb4a5e8674fe6876ccc684c3448e360087865f1704932"}
Nov 03 20:31:01.171626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.170028    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"
Nov 03 20:31:01.242151 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.214802    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"
Nov 03 20:31:01.294605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.287431    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:01.294605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.288063    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtrl6\" (UniqueName: \"kubernetes.io/projected/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e-kube-api-access-jtrl6\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.294605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.288095    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2xz9\" (UniqueName: \"kubernetes.io/projected/a370c44d-12fc-4874-a663-70e1b2fd44ad-kube-api-access-b2xz9\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.294605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.288110    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31961687-b15e-45a0-856a-874eb4baa3ec-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.294605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.288125    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a370c44d-12fc-4874-a663-70e1b2fd44ad-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.294605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.288142    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6t4w\" (UniqueName: \"kubernetes.io/projected/31961687-b15e-45a0-856a-874eb4baa3ec-kube-api-access-p6t4w\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.294605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.288157    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r59h9\" (UniqueName: \"kubernetes.io/projected/2bc004c1-979c-4f3e-ba19-fddcd5e58629-kube-api-access-r59h9\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.294605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.288170    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc004c1-979c-4f3e-ba19-fddcd5e58629-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.318358 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.317112    5659 scope.go:122] "RemoveContainer" containerID="1d12f1d96bca9eb3f7e90b44cd41a47d2e52331b1dadbd7aa26ed05a077a9742"
Nov 03 20:31:01.351473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.350772    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"
Nov 03 20:31:01.481778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.481745    5659 scope.go:122] "RemoveContainer" containerID="f01590c3c6c53d00d9999c03948f5a51e9acc4bd7f2f83bd455b6804a839ad97"
Nov 03 20:31:01.501757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.501668    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:01.509632 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.508269    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"]
Nov 03 20:31:01.513925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.513895    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-tfnfd"]
Nov 03 20:31:01.516325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.516280    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f18783ab-02f5-4154-bc33-bc738babbd40-kube-api-access-nk27q\" (UniqueName: \"kubernetes.io/projected/f18783ab-02f5-4154-bc33-bc738babbd40-kube-api-access-nk27q\") pod \"f18783ab-02f5-4154-bc33-bc738babbd40\" (UID: \"f18783ab-02f5-4154-bc33-bc738babbd40\") "
Nov 03 20:31:01.516473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.516409    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f18783ab-02f5-4154-bc33-bc738babbd40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18783ab-02f5-4154-bc33-bc738babbd40-maas-api-service-ca\") pod \"f18783ab-02f5-4154-bc33-bc738babbd40\" (UID: \"f18783ab-02f5-4154-bc33-bc738babbd40\") "
Nov 03 20:31:01.522499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.517870    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18783ab-02f5-4154-bc33-bc738babbd40-maas-api-service-ca" pod "f18783ab-02f5-4154-bc33-bc738babbd40" (UID: "f18783ab-02f5-4154-bc33-bc738babbd40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:01.682731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.664563    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18783ab-02f5-4154-bc33-bc738babbd40-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.750875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.736027    5659 scope.go:122] "RemoveContainer" containerID="dbac5ea8c3b3fe993b7f57e103fc41cb62765dc8cecf33557b1955b050825ee0"
Nov 03 20:31:01.766576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.754100    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18783ab-02f5-4154-bc33-bc738babbd40-kube-api-access-nk27q" pod "f18783ab-02f5-4154-bc33-bc738babbd40" (UID: "f18783ab-02f5-4154-bc33-bc738babbd40"). InnerVolumeSpecName "kube-api-access-nk27q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:01.930533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.924404    5659 scope.go:122] "RemoveContainer" containerID="765ab32725e9bbab5c2eb9c774779cba81707495606f64c838751f45ad26e599"
Nov 03 20:31:01.955607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.913336    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:01.955929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.955900    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"]
Nov 03 20:31:01.956101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.956089    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-kn9fh"]
Nov 03 20:31:01.960125 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.960073    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"]
Nov 03 20:31:01.962653 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.962613    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk27q\" (UniqueName: \"kubernetes.io/projected/f18783ab-02f5-4154-bc33-bc738babbd40-kube-api-access-nk27q\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:01.999234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.991160    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc5e85b-47e2-44f9-990c-7cd7780e3256" path="/var/lib/kubelet/pods/bcc5e85b-47e2-44f9-990c-7cd7780e3256/volumes"
Nov 03 20:31:01.999234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.992035    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e2428c-648d-4ef4-bed2-8b50adb40fe2" path="/var/lib/kubelet/pods/d6e2428c-648d-4ef4-bed2-8b50adb40fe2/volumes"
Nov 03 20:31:01.999234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.992925    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f154ac-29895630-mdtm5"]
Nov 03 20:31:01.999234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:01.993188    5659 scope.go:122] "RemoveContainer" containerID="202503bb3be86f37ab89740b266683337c768be32e79686b302e0bb8d2a20400"
Nov 03 20:31:02.002253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.002224    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"]
Nov 03 20:31:02.013201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.013179    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-46a6d7-29895630-kqv9d"]
Nov 03 20:31:02.027990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.027953    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"]
Nov 03 20:31:02.030981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.030954    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b503a8-29895630-272rj"]
Nov 03 20:31:02.040096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.040058    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"]
Nov 03 20:31:02.046077 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.046057    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"
Nov 03 20:31:02.046178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.046057    5659 scope.go:122] "RemoveContainer" containerID="f68f204c42c7810cc329ad56898722830ce3e8090b2d934d7dc5d831bea5ef3e"
Nov 03 20:31:02.047908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.047642    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0d4ec-29895630-qjz6v"]
Nov 03 20:31:02.055929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.055909    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"]
Nov 03 20:31:02.064235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.064069    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895630-5n6lc"]
Nov 03 20:31:02.072876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.072838    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"]
Nov 03 20:31:02.101509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.101405    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8b7351-29895630-479q8"]
Nov 03 20:31:02.110509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.110262    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:02.195355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.171345    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88035b6-4c8a-4c03-b803-e897a2c86818-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88035b6-4c8a-4c03-b803-e897a2c86818-maas-api-service-ca\") pod \"b88035b6-4c8a-4c03-b803-e897a2c86818\" (UID: \"b88035b6-4c8a-4c03-b803-e897a2c86818\") "
Nov 03 20:31:02.195355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.190456    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88035b6-4c8a-4c03-b803-e897a2c86818-kube-api-access-rmxnc\" (UniqueName: \"kubernetes.io/projected/b88035b6-4c8a-4c03-b803-e897a2c86818-kube-api-access-rmxnc\") pod \"b88035b6-4c8a-4c03-b803-e897a2c86818\" (UID: \"b88035b6-4c8a-4c03-b803-e897a2c86818\") "
Nov 03 20:31:02.261935 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.241529    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"]
Nov 03 20:31:02.261935 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.241910    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88035b6-4c8a-4c03-b803-e897a2c86818-maas-api-service-ca" pod "b88035b6-4c8a-4c03-b803-e897a2c86818" (UID: "b88035b6-4c8a-4c03-b803-e897a2c86818"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:02.261935 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.242007    5659 scope.go:122] "RemoveContainer" containerID="ef8dcbbb40cad41f8fd0f1ea5429ccdd1ad574ac875260905b06c62b68ca2790"
Nov 03 20:31:02.323607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.315086    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88035b6-4c8a-4c03-b803-e897a2c86818-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:02.392847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.389567    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:02.421967 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.410735    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895630-btkxc"]
Nov 03 20:31:02.464566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.460137    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"]
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.465901    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"]
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467194    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b88035b6-4c8a-4c03-b803-e897a2c86818" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467214    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88035b6-4c8a-4c03-b803-e897a2c86818" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467227    5659 cpu_manager.go:414] "containerMap: removing container" podUID="29399fbb-4180-4dcc-b2a0-4a18505e1fb9" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467234    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29399fbb-4180-4dcc-b2a0-4a18505e1fb9" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467255    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a370c44d-12fc-4874-a663-70e1b2fd44ad" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467263    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a370c44d-12fc-4874-a663-70e1b2fd44ad" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467278    5659 cpu_manager.go:414] "containerMap: removing container" podUID="208a1108-542a-413d-b038-9b087bb636c8" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467285    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208a1108-542a-413d-b038-9b087bb636c8" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467301    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f18783ab-02f5-4154-bc33-bc738babbd40" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467308    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f18783ab-02f5-4154-bc33-bc738babbd40" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467318    5659 cpu_manager.go:414] "containerMap: removing container" podUID="208a1108-542a-413d-b038-9b087bb636c8" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467326    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208a1108-542a-413d-b038-9b087bb636c8" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467343    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a370c44d-12fc-4874-a663-70e1b2fd44ad" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467350    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a370c44d-12fc-4874-a663-70e1b2fd44ad" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467364    5659 cpu_manager.go:414] "containerMap: removing container" podUID="31961687-b15e-45a0-856a-874eb4baa3ec" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467372    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31961687-b15e-45a0-856a-874eb4baa3ec" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467390    5659 cpu_manager.go:414] "containerMap: removing container" podUID="31961687-b15e-45a0-856a-874eb4baa3ec" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467396    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31961687-b15e-45a0-856a-874eb4baa3ec" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467433    5659 cpu_manager.go:414] "containerMap: removing container" podUID="2bc004c1-979c-4f3e-ba19-fddcd5e58629" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467440    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc004c1-979c-4f3e-ba19-fddcd5e58629" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467458    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b88035b6-4c8a-4c03-b803-e897a2c86818" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467465    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88035b6-4c8a-4c03-b803-e897a2c86818" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467477    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c1c13e2c-aeb0-4eb2-90ed-78c91e40280e" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467499    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c13e2c-aeb0-4eb2-90ed-78c91e40280e" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467545    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c1c13e2c-aeb0-4eb2-90ed-78c91e40280e" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467552    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c13e2c-aeb0-4eb2-90ed-78c91e40280e" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467565    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f18783ab-02f5-4154-bc33-bc738babbd40" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467574    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f18783ab-02f5-4154-bc33-bc738babbd40" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467759    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88035b6-4c8a-4c03-b803-e897a2c86818" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467771    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc004c1-979c-4f3e-ba19-fddcd5e58629" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467782    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c13e2c-aeb0-4eb2-90ed-78c91e40280e" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467790    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f18783ab-02f5-4154-bc33-bc738babbd40" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467800    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88035b6-4c8a-4c03-b803-e897a2c86818" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467811    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="29399fbb-4180-4dcc-b2a0-4a18505e1fb9" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467822    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a370c44d-12fc-4874-a663-70e1b2fd44ad" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467830    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="208a1108-542a-413d-b038-9b087bb636c8" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467840    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f18783ab-02f5-4154-bc33-bc738babbd40" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467851    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="208a1108-542a-413d-b038-9b087bb636c8" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467863    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a370c44d-12fc-4874-a663-70e1b2fd44ad" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467872    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="29399fbb-4180-4dcc-b2a0-4a18505e1fb9" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467882    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="31961687-b15e-45a0-856a-874eb4baa3ec" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467892    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="31961687-b15e-45a0-856a-874eb4baa3ec" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467905    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc004c1-979c-4f3e-ba19-fddcd5e58629" containerName="cleanup"
Nov 03 20:31:02.470352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.467917    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f938b051-c9cc-4600-89b3-f2a4fde514ec" containerName="cleanup"
Nov 03 20:31:02.478354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.477602    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"
Nov 03 20:31:02.490727 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.490694    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"]
Nov 03 20:31:02.501850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.500988    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l" event={"ID":"b88035b6-4c8a-4c03-b803-e897a2c86818","Type":"ContainerDied","Data":"e470143f18f63dc61a78b15c4943b7051ffadddfd4b6f849b4d63b40fb6af387"}
Nov 03 20:31:02.501850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.501116    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"
Nov 03 20:31:02.503642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.502629    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88035b6-4c8a-4c03-b803-e897a2c86818-kube-api-access-rmxnc" pod "b88035b6-4c8a-4c03-b803-e897a2c86818" (UID: "b88035b6-4c8a-4c03-b803-e897a2c86818"). InnerVolumeSpecName "kube-api-access-rmxnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:02.513094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.512886    5659 generic.go:356] "Generic (PLEG): container finished" podID="9e43233f-4023-428a-8cff-fffd9ab19d74" containerID="9bade9ba2fc56e6228b95c27f46f07c600f7e6687752ed706ed891f8e4670b66" exitCode=60
Nov 03 20:31:02.513094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.512983    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw" event={"ID":"9e43233f-4023-428a-8cff-fffd9ab19d74","Type":"ContainerDied","Data":"9bade9ba2fc56e6228b95c27f46f07c600f7e6687752ed706ed891f8e4670b66"}
Nov 03 20:31:02.518863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.518759    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks" event={"ID":"f18783ab-02f5-4154-bc33-bc738babbd40","Type":"ContainerDied","Data":"f5906459ea21e64ca99b7fca357cc9759e4fa18a370c377ffcbd18dfef042f85"}
Nov 03 20:31:02.519618 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.519128    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"
Nov 03 20:31:02.529115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.525988    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99" event={"ID":"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5","Type":"ContainerStarted","Data":"e0a052fc7209dd388c51628222fd0d5407ab1026176ceb596d4ba6b1ffe9a535"}
Nov 03 20:31:02.529115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.526142    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99" podUID="8f042faa-25c7-4ce0-9fb8-16fc7633f5d5" containerName="cleanup" containerID="cri-o://e0a052fc7209dd388c51628222fd0d5407ab1026176ceb596d4ba6b1ffe9a535" gracePeriod=30
Nov 03 20:31:02.529115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.526378    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmxnc\" (UniqueName: \"kubernetes.io/projected/b88035b6-4c8a-4c03-b803-e897a2c86818-kube-api-access-rmxnc\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:02.552309 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:31:02.549121    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0005549b_99f3_4bd9_b80d_3414d16e1389.slice/crio-70c1859d1bf43df4baa583ae71302bce744f326386cac03a4d1764be71f32ffe WatchSource:0}: Error finding container 70c1859d1bf43df4baa583ae71302bce744f326386cac03a4d1764be71f32ffe: Status 404 returned error can't find the container with id 70c1859d1bf43df4baa583ae71302bce744f326386cac03a4d1764be71f32ffe
Nov 03 20:31:02.552309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.551950    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"]
Nov 03 20:31:02.582575 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.582547    5659 scope.go:122] "RemoveContainer" containerID="e83f41bcae8b8894f2c9e53468d1c5eb8a6ffca9108f56551d22e419c0c0a3cc"
Nov 03 20:31:02.602345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.602325    5659 scope.go:122] "RemoveContainer" containerID="5bf06c691ffc7c029814523c054899ca996a85f5e5f781bd7bc98becc342f263"
Nov 03 20:31:02.607702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.607677    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"
Nov 03 20:31:02.612377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.612348    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"]
Nov 03 20:31:02.620324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.620296    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-32e482-29895630-6svks"]
Nov 03 20:31:02.625098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.625073    5659 scope.go:122] "RemoveContainer" containerID="48c93a6115111b023f0e77ac5fe136d48cb2ffddd34331eb469886a346d29af8"
Nov 03 20:31:02.628296 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.628269    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-6frs2\" (UID: \"3bf82ad8-facf-49ca-a847-65af5a5f4154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"
Nov 03 20:31:02.628556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.628540    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-6frs2\" (UID: \"3bf82ad8-facf-49ca-a847-65af5a5f4154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"
Nov 03 20:31:02.628757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.628730    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5jk2\" (UniqueName: \"kubernetes.io/projected/3bf82ad8-facf-49ca-a847-65af5a5f4154-kube-api-access-t5jk2\") pod \"maas-api-revoke-sub-f14db2bd-6frs2\" (UID: \"3bf82ad8-facf-49ca-a847-65af5a5f4154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"
Nov 03 20:31:02.641930 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.641906    5659 scope.go:122] "RemoveContainer" containerID="dad3ee195739fad2412a2832a8ae3ca6a545960f3bbf7ef8025dafee17444b38"
Nov 03 20:31:02.729963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.729926    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e43233f-4023-428a-8cff-fffd9ab19d74-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e43233f-4023-428a-8cff-fffd9ab19d74-maas-api-service-ca\") pod \"9e43233f-4023-428a-8cff-fffd9ab19d74\" (UID: \"9e43233f-4023-428a-8cff-fffd9ab19d74\") "
Nov 03 20:31:02.730394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.730275    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e43233f-4023-428a-8cff-fffd9ab19d74-kube-api-access-blqgv\" (UniqueName: \"kubernetes.io/projected/9e43233f-4023-428a-8cff-fffd9ab19d74-kube-api-access-blqgv\") pod \"9e43233f-4023-428a-8cff-fffd9ab19d74\" (UID: \"9e43233f-4023-428a-8cff-fffd9ab19d74\") "
Nov 03 20:31:02.730394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.730351    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e43233f-4023-428a-8cff-fffd9ab19d74-maas-api-service-ca" pod "9e43233f-4023-428a-8cff-fffd9ab19d74" (UID: "9e43233f-4023-428a-8cff-fffd9ab19d74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:02.730809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.730788    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-6frs2\" (UID: \"3bf82ad8-facf-49ca-a847-65af5a5f4154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"
Nov 03 20:31:02.730928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.730914    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5jk2\" (UniqueName: \"kubernetes.io/projected/3bf82ad8-facf-49ca-a847-65af5a5f4154-kube-api-access-t5jk2\") pod \"maas-api-revoke-sub-f14db2bd-6frs2\" (UID: \"3bf82ad8-facf-49ca-a847-65af5a5f4154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"
Nov 03 20:31:02.731104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.731085    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-6frs2\" (UID: \"3bf82ad8-facf-49ca-a847-65af5a5f4154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"
Nov 03 20:31:02.731335 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.731319    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e43233f-4023-428a-8cff-fffd9ab19d74-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:02.733360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.733288    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-6frs2\" (UID: \"3bf82ad8-facf-49ca-a847-65af5a5f4154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"
Nov 03 20:31:02.737564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.737223    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-6frs2\" (UID: \"3bf82ad8-facf-49ca-a847-65af5a5f4154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"
Nov 03 20:31:02.739251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.738675    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e43233f-4023-428a-8cff-fffd9ab19d74-kube-api-access-blqgv" pod "9e43233f-4023-428a-8cff-fffd9ab19d74" (UID: "9e43233f-4023-428a-8cff-fffd9ab19d74"). InnerVolumeSpecName "kube-api-access-blqgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:02.746959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.746928    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5jk2\" (UniqueName: \"kubernetes.io/projected/3bf82ad8-facf-49ca-a847-65af5a5f4154-kube-api-access-t5jk2\") pod \"maas-api-revoke-sub-f14db2bd-6frs2\" (UID: \"3bf82ad8-facf-49ca-a847-65af5a5f4154\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"
Nov 03 20:31:02.832111 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.832038    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"]
Nov 03 20:31:02.837071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.837015    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blqgv\" (UniqueName: \"kubernetes.io/projected/9e43233f-4023-428a-8cff-fffd9ab19d74-kube-api-access-blqgv\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:02.849182 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.849142    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:02.852470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.852439    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-81a5ae-29895630-7s87l"]
Nov 03 20:31:02.905501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.905216    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"
Nov 03 20:31:02.960431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.960402    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"
Nov 03 20:31:02.989617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:02.989582    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:03.083903 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.062327    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-kube-api-access-zglvx\" (UniqueName: \"kubernetes.io/projected/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-kube-api-access-zglvx\") pod \"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5\" (UID: \"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5\") "
Nov 03 20:31:03.137567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.127814    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-kube-api-access-zglvx" pod "8f042faa-25c7-4ce0-9fb8-16fc7633f5d5" (UID: "8f042faa-25c7-4ce0-9fb8-16fc7633f5d5"). InnerVolumeSpecName "kube-api-access-zglvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:03.140658 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.140625    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zglvx\" (UniqueName: \"kubernetes.io/projected/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-kube-api-access-zglvx\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:03.273557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.273520    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-maas-api-service-ca\") pod \"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5\" (UID: \"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5\") "
Nov 03 20:31:03.368726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.368670    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-maas-api-service-ca" pod "8f042faa-25c7-4ce0-9fb8-16fc7633f5d5" (UID: "8f042faa-25c7-4ce0-9fb8-16fc7633f5d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:03.383650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.383607    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5-maas-api-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:03.450882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.450848    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:03.484324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.484288    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"]
Nov 03 20:31:03.543178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.543147    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw" event={"ID":"9e43233f-4023-428a-8cff-fffd9ab19d74","Type":"ContainerDied","Data":"5ebc0bf36ed010a6294575570fc70d49ddbb8779a9f1c05a444b6e902990a0d0"}
Nov 03 20:31:03.543334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.543218    5659 scope.go:122] "RemoveContainer" containerID="9bade9ba2fc56e6228b95c27f46f07c600f7e6687752ed706ed891f8e4670b66"
Nov 03 20:31:03.543334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.543156    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"
Nov 03 20:31:03.546082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.546053    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2" event={"ID":"3bf82ad8-facf-49ca-a847-65af5a5f4154","Type":"ContainerStarted","Data":"afb7c10e212005eb4cf6f968862a1d4da762a0f5b5476cc1aa519700787f0926"}
Nov 03 20:31:03.548116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.548096    5659 generic.go:356] "Generic (PLEG): container finished" podID="8f042faa-25c7-4ce0-9fb8-16fc7633f5d5" containerID="e0a052fc7209dd388c51628222fd0d5407ab1026176ceb596d4ba6b1ffe9a535" exitCode=60
Nov 03 20:31:03.548216 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.548139    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"
Nov 03 20:31:03.548216 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.548172    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99" event={"ID":"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5","Type":"ContainerDied","Data":"e0a052fc7209dd388c51628222fd0d5407ab1026176ceb596d4ba6b1ffe9a535"}
Nov 03 20:31:03.548331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.548267    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99" event={"ID":"8f042faa-25c7-4ce0-9fb8-16fc7633f5d5","Type":"ContainerDied","Data":"ba3b9281b98f4c4a3ba8d291a6d7f4bd0476f472e94be2ddae4c635288ff9113"}
Nov 03 20:31:03.554338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.554308    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b" event={"ID":"0005549b-99f3-4bd9-b80d-3414d16e1389","Type":"ContainerStarted","Data":"e20e121e04198e615b439b4234d1b5ae89935696e038944f3f300436e6bf133b"}
Nov 03 20:31:03.554443 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.554363    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b" event={"ID":"0005549b-99f3-4bd9-b80d-3414d16e1389","Type":"ContainerStarted","Data":"70c1859d1bf43df4baa583ae71302bce744f326386cac03a4d1764be71f32ffe"}
Nov 03 20:31:03.556390 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.556354    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw" event={"ID":"1c1247bb-a71b-4f74-8261-c56a31e13462","Type":"ContainerStarted","Data":"b7d2b9af5dc954f4884f4e9c4145c058b620df1098b915cea46662589bf54478"}
Nov 03 20:31:03.556558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.556412    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw" event={"ID":"1c1247bb-a71b-4f74-8261-c56a31e13462","Type":"ContainerStarted","Data":"ed423095cc327dc77e2d26566bf6d83965bd7f129f221a216f9e4084b9963d55"}
Nov 03 20:31:03.557742 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.557719    5659 scope.go:122] "RemoveContainer" containerID="e0a052fc7209dd388c51628222fd0d5407ab1026176ceb596d4ba6b1ffe9a535"
Nov 03 20:31:03.572079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.572051    5659 scope.go:122] "RemoveContainer" containerID="c44b88e61ac20fa7323572a0436e270afee146992d826d8b3836b5523af01330"
Nov 03 20:31:03.583340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.583313    5659 scope.go:122] "RemoveContainer" containerID="e0a052fc7209dd388c51628222fd0d5407ab1026176ceb596d4ba6b1ffe9a535"
Nov 03 20:31:03.583671 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:03.583650    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a052fc7209dd388c51628222fd0d5407ab1026176ceb596d4ba6b1ffe9a535\": container with ID starting with e0a052fc7209dd388c51628222fd0d5407ab1026176ceb596d4ba6b1ffe9a535 not found: ID does not exist" containerID="e0a052fc7209dd388c51628222fd0d5407ab1026176ceb596d4ba6b1ffe9a535"
Nov 03 20:31:03.583718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.583680    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a052fc7209dd388c51628222fd0d5407ab1026176ceb596d4ba6b1ffe9a535"} err="failed to get container status \"e0a052fc7209dd388c51628222fd0d5407ab1026176ceb596d4ba6b1ffe9a535\": rpc error: code = NotFound desc = could not find container \"e0a052fc7209dd388c51628222fd0d5407ab1026176ceb596d4ba6b1ffe9a535\": container with ID starting with e0a052fc7209dd388c51628222fd0d5407ab1026176ceb596d4ba6b1ffe9a535 not found: ID does not exist"
Nov 03 20:31:03.583718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.583701    5659 scope.go:122] "RemoveContainer" containerID="c44b88e61ac20fa7323572a0436e270afee146992d826d8b3836b5523af01330"
Nov 03 20:31:03.583995 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:03.583970    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44b88e61ac20fa7323572a0436e270afee146992d826d8b3836b5523af01330\": container with ID starting with c44b88e61ac20fa7323572a0436e270afee146992d826d8b3836b5523af01330 not found: ID does not exist" containerID="c44b88e61ac20fa7323572a0436e270afee146992d826d8b3836b5523af01330"
Nov 03 20:31:03.584031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.584002    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44b88e61ac20fa7323572a0436e270afee146992d826d8b3836b5523af01330"} err="failed to get container status \"c44b88e61ac20fa7323572a0436e270afee146992d826d8b3836b5523af01330\": rpc error: code = NotFound desc = could not find container \"c44b88e61ac20fa7323572a0436e270afee146992d826d8b3836b5523af01330\": container with ID starting with c44b88e61ac20fa7323572a0436e270afee146992d826d8b3836b5523af01330 not found: ID does not exist"
Nov 03 20:31:03.590056 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.590004    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw" podStartSLOduration=62.589988368 podStartE2EDuration="1m2.589988368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:03.588901457 +0000 UTC m=+3215.743803877" watchObservedRunningTime="2026-11-03 20:31:03.589988368 +0000 UTC m=+3215.744890789"
Nov 03 20:31:03.608527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.608429    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"]
Nov 03 20:31:03.613637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.613595    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e4ce7-29895630-ddm99"]
Nov 03 20:31:03.624100 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.624052    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"]
Nov 03 20:31:03.629460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.629426    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-33e267-29895630-4b4pw"]
Nov 03 20:31:03.644382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.644314    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b" podStartSLOduration=61.644295644 podStartE2EDuration="1m1.644295644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:03.64344574 +0000 UTC m=+3215.798348175" watchObservedRunningTime="2026-11-03 20:31:03.644295644 +0000 UTC m=+3215.799198064"
Nov 03 20:31:03.800433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.800397    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:03.962236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.962198    5659 scope.go:122] "RemoveContainer" containerID="8a7287031f0bbdfb6c6a89b349cf9d7a37acae17aec29d066d91c7a32bdd338f"
Nov 03 20:31:03.973951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.973921    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208a1108-542a-413d-b038-9b087bb636c8" path="/var/lib/kubelet/pods/208a1108-542a-413d-b038-9b087bb636c8/volumes"
Nov 03 20:31:03.974562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.974536    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29399fbb-4180-4dcc-b2a0-4a18505e1fb9" path="/var/lib/kubelet/pods/29399fbb-4180-4dcc-b2a0-4a18505e1fb9/volumes"
Nov 03 20:31:03.975177 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.975158    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc004c1-979c-4f3e-ba19-fddcd5e58629" path="/var/lib/kubelet/pods/2bc004c1-979c-4f3e-ba19-fddcd5e58629/volumes"
Nov 03 20:31:03.976176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.976155    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31961687-b15e-45a0-856a-874eb4baa3ec" path="/var/lib/kubelet/pods/31961687-b15e-45a0-856a-874eb4baa3ec/volumes"
Nov 03 20:31:03.976741 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.976721    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f042faa-25c7-4ce0-9fb8-16fc7633f5d5" path="/var/lib/kubelet/pods/8f042faa-25c7-4ce0-9fb8-16fc7633f5d5/volumes"
Nov 03 20:31:03.977179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.977163    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e43233f-4023-428a-8cff-fffd9ab19d74" path="/var/lib/kubelet/pods/9e43233f-4023-428a-8cff-fffd9ab19d74/volumes"
Nov 03 20:31:03.977924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.977908    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a370c44d-12fc-4874-a663-70e1b2fd44ad" path="/var/lib/kubelet/pods/a370c44d-12fc-4874-a663-70e1b2fd44ad/volumes"
Nov 03 20:31:03.978405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.978387    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88035b6-4c8a-4c03-b803-e897a2c86818" path="/var/lib/kubelet/pods/b88035b6-4c8a-4c03-b803-e897a2c86818/volumes"
Nov 03 20:31:03.979040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.979021    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c13e2c-aeb0-4eb2-90ed-78c91e40280e" path="/var/lib/kubelet/pods/c1c13e2c-aeb0-4eb2-90ed-78c91e40280e/volumes"
Nov 03 20:31:03.980094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.980072    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f18783ab-02f5-4154-bc33-bc738babbd40" path="/var/lib/kubelet/pods/f18783ab-02f5-4154-bc33-bc738babbd40/volumes"
Nov 03 20:31:03.980729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:03.980703    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f938b051-c9cc-4600-89b3-f2a4fde514ec" path="/var/lib/kubelet/pods/f938b051-c9cc-4600-89b3-f2a4fde514ec/volumes"
Nov 03 20:31:04.598624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:04.595715    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:04.648328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:04.648298    5659 generic.go:356] "Generic (PLEG): container finished" podID="0005549b-99f3-4bd9-b80d-3414d16e1389" containerID="e20e121e04198e615b439b4234d1b5ae89935696e038944f3f300436e6bf133b" exitCode=22
Nov 03 20:31:04.648563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:04.648383    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b" event={"ID":"0005549b-99f3-4bd9-b80d-3414d16e1389","Type":"ContainerDied","Data":"e20e121e04198e615b439b4234d1b5ae89935696e038944f3f300436e6bf133b"}
Nov 03 20:31:04.648777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:04.648758    5659 scope.go:122] "RemoveContainer" containerID="e20e121e04198e615b439b4234d1b5ae89935696e038944f3f300436e6bf133b"
Nov 03 20:31:04.651049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:04.651024    5659 generic.go:356] "Generic (PLEG): container finished" podID="1c1247bb-a71b-4f74-8261-c56a31e13462" containerID="b7d2b9af5dc954f4884f4e9c4145c058b620df1098b915cea46662589bf54478" exitCode=7
Nov 03 20:31:04.651162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:04.651140    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw" event={"ID":"1c1247bb-a71b-4f74-8261-c56a31e13462","Type":"ContainerDied","Data":"b7d2b9af5dc954f4884f4e9c4145c058b620df1098b915cea46662589bf54478"}
Nov 03 20:31:04.651493 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:04.651375    5659 scope.go:122] "RemoveContainer" containerID="b7d2b9af5dc954f4884f4e9c4145c058b620df1098b915cea46662589bf54478"
Nov 03 20:31:04.662939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:04.662903    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2" event={"ID":"3bf82ad8-facf-49ca-a847-65af5a5f4154","Type":"ContainerStarted","Data":"5519aa9dc2f7d16eec21c0fd5d1258a5d704c54bee6cf6ec857247d5fd11a051"}
Nov 03 20:31:04.699911 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:04.699857    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2" podStartSLOduration=2.699842658 podStartE2EDuration="2.699842658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:04.687938941 +0000 UTC m=+3216.842841384" watchObservedRunningTime="2026-11-03 20:31:04.699842658 +0000 UTC m=+3216.854745076"
Nov 03 20:31:04.962237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:04.962209    5659 scope.go:122] "RemoveContainer" containerID="58366ccee1839a736011290888bb84541c5b61bdc9fb87c0a637d25c705ea5d3"
Nov 03 20:31:05.273652 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:05.273445    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:05.691096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:05.690996    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b" event={"ID":"0005549b-99f3-4bd9-b80d-3414d16e1389","Type":"ContainerStarted","Data":"d27628f2851bd263cde76b03586aea89f5ad7de9a54afce019e7efa9b6aa5e45"}
Nov 03 20:31:05.700165 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:05.700131    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw" event={"ID":"1c1247bb-a71b-4f74-8261-c56a31e13462","Type":"ContainerStarted","Data":"c9ac06ca0f2c26bb52caa86d4ccfa94541f56af86bcf9dad54af8efb7d8447cc"}
Nov 03 20:31:05.705998 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:05.705961    5659 generic.go:356] "Generic (PLEG): container finished" podID="3bf82ad8-facf-49ca-a847-65af5a5f4154" containerID="5519aa9dc2f7d16eec21c0fd5d1258a5d704c54bee6cf6ec857247d5fd11a051" exitCode=22
Nov 03 20:31:05.706163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:05.705998    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2" event={"ID":"3bf82ad8-facf-49ca-a847-65af5a5f4154","Type":"ContainerDied","Data":"5519aa9dc2f7d16eec21c0fd5d1258a5d704c54bee6cf6ec857247d5fd11a051"}
Nov 03 20:31:05.706359 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:05.706341    5659 scope.go:122] "RemoveContainer" containerID="5519aa9dc2f7d16eec21c0fd5d1258a5d704c54bee6cf6ec857247d5fd11a051"
Nov 03 20:31:05.710781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:05.710740    5659 generic.go:356] "Generic (PLEG): container finished" podID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" containerID="1b42781e0f0b728f0d611873df4f27b9bca3729d88ea0f3c5e71ac3103214bf8" exitCode=22
Nov 03 20:31:05.710926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:05.710813    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c" event={"ID":"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b","Type":"ContainerDied","Data":"1b42781e0f0b728f0d611873df4f27b9bca3729d88ea0f3c5e71ac3103214bf8"}
Nov 03 20:31:05.710926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:05.710885    5659 scope.go:122] "RemoveContainer" containerID="8a7287031f0bbdfb6c6a89b349cf9d7a37acae17aec29d066d91c7a32bdd338f"
Nov 03 20:31:05.711842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:05.711821    5659 scope.go:122] "RemoveContainer" containerID="1b42781e0f0b728f0d611873df4f27b9bca3729d88ea0f3c5e71ac3103214bf8"
Nov 03 20:31:05.712744 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:05.712720    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-kw64c_odh-ai-gateway-infra(20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c" podUID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b"
Nov 03 20:31:05.716103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:05.716072    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8" event={"ID":"c6f1d55f-acb8-45e5-bffe-96f1937c0d64","Type":"ContainerStarted","Data":"09263df3349fb96465c172dd4432550cf30de73e9ebac7c7ffb3c4f4b86890f3"}
Nov 03 20:31:06.120809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.120768    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:06.728925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.728880    5659 generic.go:356] "Generic (PLEG): container finished" podID="0005549b-99f3-4bd9-b80d-3414d16e1389" containerID="d27628f2851bd263cde76b03586aea89f5ad7de9a54afce019e7efa9b6aa5e45" exitCode=22
Nov 03 20:31:06.729116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.728961    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b" event={"ID":"0005549b-99f3-4bd9-b80d-3414d16e1389","Type":"ContainerDied","Data":"d27628f2851bd263cde76b03586aea89f5ad7de9a54afce019e7efa9b6aa5e45"}
Nov 03 20:31:06.729116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.729028    5659 scope.go:122] "RemoveContainer" containerID="e20e121e04198e615b439b4234d1b5ae89935696e038944f3f300436e6bf133b"
Nov 03 20:31:06.729468 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.729450    5659 scope.go:122] "RemoveContainer" containerID="d27628f2851bd263cde76b03586aea89f5ad7de9a54afce019e7efa9b6aa5e45"
Nov 03 20:31:06.730028 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:06.729998    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-fmd6b_odh-ai-gateway-infra(0005549b-99f3-4bd9-b80d-3414d16e1389)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b" podUID="0005549b-99f3-4bd9-b80d-3414d16e1389"
Nov 03 20:31:06.731576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.731549    5659 generic.go:356] "Generic (PLEG): container finished" podID="1c1247bb-a71b-4f74-8261-c56a31e13462" containerID="c9ac06ca0f2c26bb52caa86d4ccfa94541f56af86bcf9dad54af8efb7d8447cc" exitCode=7
Nov 03 20:31:06.731905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.731886    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw" event={"ID":"1c1247bb-a71b-4f74-8261-c56a31e13462","Type":"ContainerDied","Data":"c9ac06ca0f2c26bb52caa86d4ccfa94541f56af86bcf9dad54af8efb7d8447cc"}
Nov 03 20:31:06.732143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.732125    5659 scope.go:122] "RemoveContainer" containerID="c9ac06ca0f2c26bb52caa86d4ccfa94541f56af86bcf9dad54af8efb7d8447cc"
Nov 03 20:31:06.732601 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:06.732574    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-knsvw_odh-ai-gateway-infra(1c1247bb-a71b-4f74-8261-c56a31e13462)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw" podUID="1c1247bb-a71b-4f74-8261-c56a31e13462"
Nov 03 20:31:06.734660 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.734635    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2" event={"ID":"3bf82ad8-facf-49ca-a847-65af5a5f4154","Type":"ContainerStarted","Data":"19d6c4413809f3cbee328e9f68146c18c1556a819983f4a265e40672faab6222"}
Nov 03 20:31:06.739949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.739925    5659 generic.go:356] "Generic (PLEG): container finished" podID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64" containerID="09263df3349fb96465c172dd4432550cf30de73e9ebac7c7ffb3c4f4b86890f3" exitCode=22
Nov 03 20:31:06.740057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.739997    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8" event={"ID":"c6f1d55f-acb8-45e5-bffe-96f1937c0d64","Type":"ContainerDied","Data":"09263df3349fb96465c172dd4432550cf30de73e9ebac7c7ffb3c4f4b86890f3"}
Nov 03 20:31:06.740456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.740437    5659 scope.go:122] "RemoveContainer" containerID="09263df3349fb96465c172dd4432550cf30de73e9ebac7c7ffb3c4f4b86890f3"
Nov 03 20:31:06.740786 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:06.740765    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-6z5q8_odh-ai-gateway-infra(c6f1d55f-acb8-45e5-bffe-96f1937c0d64)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8" podUID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64"
Nov 03 20:31:06.745993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.745974    5659 scope.go:122] "RemoveContainer" containerID="b7d2b9af5dc954f4884f4e9c4145c058b620df1098b915cea46662589bf54478"
Nov 03 20:31:06.763132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.763016    5659 scope.go:122] "RemoveContainer" containerID="58366ccee1839a736011290888bb84541c5b61bdc9fb87c0a637d25c705ea5d3"
Nov 03 20:31:06.768458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.768432    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"]
Nov 03 20:31:06.817398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.817361    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"]
Nov 03 20:31:06.954013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.953542    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:06.969224 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:06.964415    5659 scope.go:122] "RemoveContainer" containerID="07ac5c968ed7900c1dc38b78c7f3ee5783c27e8e0b752dbba411fcfa4e85e08f"
Nov 03 20:31:07.096826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.096802    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"
Nov 03 20:31:07.264127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.264093    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-kube-api-access-g4hq7\" (UniqueName: \"kubernetes.io/projected/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-kube-api-access-g4hq7\") pod \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\" (UID: \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\") "
Nov 03 20:31:07.264604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.264281    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-openshift-ca\") pod \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\" (UID: \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\") "
Nov 03 20:31:07.264604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.264409    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-xks-ca\") pod \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\" (UID: \"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b\") "
Nov 03 20:31:07.264726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.264702    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-openshift-ca" pod "20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" (UID: "20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:07.265291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.265268    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:07.266309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.266283    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-kube-api-access-g4hq7" pod "20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" (UID: "20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b"). InnerVolumeSpecName "kube-api-access-g4hq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:07.266405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.266352    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-xks-ca" pod "20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" (UID: "20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:07.367886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.367844    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4hq7\" (UniqueName: \"kubernetes.io/projected/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-kube-api-access-g4hq7\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:07.367886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.367890    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:07.701883 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.701803    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:07.823297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.823261    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj" event={"ID":"56bf2d77-4688-4756-a9bb-01f7b346457c","Type":"ContainerStarted","Data":"a176131294668e220f92409dfd9c63eef888b90da3613ca7134c81001e983c40"}
Nov 03 20:31:07.826286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.825909    5659 scope.go:122] "RemoveContainer" containerID="d27628f2851bd263cde76b03586aea89f5ad7de9a54afce019e7efa9b6aa5e45"
Nov 03 20:31:07.826605 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:07.826566    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-fmd6b_odh-ai-gateway-infra(0005549b-99f3-4bd9-b80d-3414d16e1389)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b" podUID="0005549b-99f3-4bd9-b80d-3414d16e1389"
Nov 03 20:31:07.828423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.828405    5659 scope.go:122] "RemoveContainer" containerID="c9ac06ca0f2c26bb52caa86d4ccfa94541f56af86bcf9dad54af8efb7d8447cc"
Nov 03 20:31:07.828780 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:07.828753    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-knsvw_odh-ai-gateway-infra(1c1247bb-a71b-4f74-8261-c56a31e13462)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw" podUID="1c1247bb-a71b-4f74-8261-c56a31e13462"
Nov 03 20:31:07.830772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.830745    5659 generic.go:356] "Generic (PLEG): container finished" podID="3bf82ad8-facf-49ca-a847-65af5a5f4154" containerID="19d6c4413809f3cbee328e9f68146c18c1556a819983f4a265e40672faab6222" exitCode=22
Nov 03 20:31:07.830917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.830895    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2" event={"ID":"3bf82ad8-facf-49ca-a847-65af5a5f4154","Type":"ContainerDied","Data":"19d6c4413809f3cbee328e9f68146c18c1556a819983f4a265e40672faab6222"}
Nov 03 20:31:07.830988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.830955    5659 scope.go:122] "RemoveContainer" containerID="5519aa9dc2f7d16eec21c0fd5d1258a5d704c54bee6cf6ec857247d5fd11a051"
Nov 03 20:31:07.831762 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.831101    5659 scope.go:122] "RemoveContainer" containerID="19d6c4413809f3cbee328e9f68146c18c1556a819983f4a265e40672faab6222"
Nov 03 20:31:07.831762 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:07.831500    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-6frs2_odh-ai-gateway-infra(3bf82ad8-facf-49ca-a847-65af5a5f4154)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2" podUID="3bf82ad8-facf-49ca-a847-65af5a5f4154"
Nov 03 20:31:07.833356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.833342    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"
Nov 03 20:31:07.833439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.833369    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c" event={"ID":"20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b","Type":"ContainerDied","Data":"ff933c6135c4e713f295aad091b2ae0353999a8f530a5efc6b978380ba11b9f1"}
Nov 03 20:31:07.854657 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.854633    5659 scope.go:122] "RemoveContainer" containerID="1b42781e0f0b728f0d611873df4f27b9bca3729d88ea0f3c5e71ac3103214bf8"
Nov 03 20:31:07.956755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.956264    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"]
Nov 03 20:31:07.980272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:07.980238    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-kw64c"]
Nov 03 20:31:08.013552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.013477    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"
Nov 03 20:31:08.123849 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.123818    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-kube-api-access-f7b4w\" (UniqueName: \"kubernetes.io/projected/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-kube-api-access-f7b4w\") pod \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\" (UID: \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\") "
Nov 03 20:31:08.124062 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.123869    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-openshift-ca\") pod \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\" (UID: \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\") "
Nov 03 20:31:08.124062 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.123888    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-xks-ca\") pod \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\" (UID: \"c6f1d55f-acb8-45e5-bffe-96f1937c0d64\") "
Nov 03 20:31:08.124368 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.124341    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-openshift-ca" pod "c6f1d55f-acb8-45e5-bffe-96f1937c0d64" (UID: "c6f1d55f-acb8-45e5-bffe-96f1937c0d64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:08.124963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.124937    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:08.126400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.126264    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-xks-ca" pod "c6f1d55f-acb8-45e5-bffe-96f1937c0d64" (UID: "c6f1d55f-acb8-45e5-bffe-96f1937c0d64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:08.126400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.126378    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-kube-api-access-f7b4w" pod "c6f1d55f-acb8-45e5-bffe-96f1937c0d64" (UID: "c6f1d55f-acb8-45e5-bffe-96f1937c0d64"). InnerVolumeSpecName "kube-api-access-f7b4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:08.227583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.227491    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7b4w\" (UniqueName: \"kubernetes.io/projected/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-kube-api-access-f7b4w\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:08.227583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.227524    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f1d55f-acb8-45e5-bffe-96f1937c0d64-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:08.587441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.587359    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:08.644609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.644463    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"]
Nov 03 20:31:08.776892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.776853    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:08.847105 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.847024    5659 generic.go:356] "Generic (PLEG): container finished" podID="56bf2d77-4688-4756-a9bb-01f7b346457c" containerID="a176131294668e220f92409dfd9c63eef888b90da3613ca7134c81001e983c40" exitCode=22
Nov 03 20:31:08.847277 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.847099    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj" event={"ID":"56bf2d77-4688-4756-a9bb-01f7b346457c","Type":"ContainerDied","Data":"a176131294668e220f92409dfd9c63eef888b90da3613ca7134c81001e983c40"}
Nov 03 20:31:08.847277 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.847175    5659 scope.go:122] "RemoveContainer" containerID="07ac5c968ed7900c1dc38b78c7f3ee5783c27e8e0b752dbba411fcfa4e85e08f"
Nov 03 20:31:08.847671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.847651    5659 scope.go:122] "RemoveContainer" containerID="a176131294668e220f92409dfd9c63eef888b90da3613ca7134c81001e983c40"
Nov 03 20:31:08.848010 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:08.847980    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-tbfnj_odh-ai-gateway-infra(56bf2d77-4688-4756-a9bb-01f7b346457c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj" podUID="56bf2d77-4688-4756-a9bb-01f7b346457c"
Nov 03 20:31:08.849551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.849531    5659 scope.go:122] "RemoveContainer" containerID="19d6c4413809f3cbee328e9f68146c18c1556a819983f4a265e40672faab6222"
Nov 03 20:31:08.849818 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:08.849787    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-6frs2_odh-ai-gateway-infra(3bf82ad8-facf-49ca-a847-65af5a5f4154)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2" podUID="3bf82ad8-facf-49ca-a847-65af5a5f4154"
Nov 03 20:31:08.852215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.852194    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"
Nov 03 20:31:08.852306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.852223    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8" event={"ID":"c6f1d55f-acb8-45e5-bffe-96f1937c0d64","Type":"ContainerDied","Data":"c62793d118a1ff2807351c0a0033f47ad1c6de5eb139358f571afdcece8126b5"}
Nov 03 20:31:08.864933 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.864911    5659 scope.go:122] "RemoveContainer" containerID="09263df3349fb96465c172dd4432550cf30de73e9ebac7c7ffb3c4f4b86890f3"
Nov 03 20:31:08.871882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.871857    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"]
Nov 03 20:31:08.993514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:08.958262    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:09.304088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:09.288576    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"]
Nov 03 20:31:09.314535 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:09.313492    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-6z5q8"]
Nov 03 20:31:09.453426 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:09.453369    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:09.980501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:09.979648    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:09.997654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:09.997606    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" path="/var/lib/kubelet/pods/20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b/volumes"
Nov 03 20:31:09.998461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:09.998435    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64" path="/var/lib/kubelet/pods/c6f1d55f-acb8-45e5-bffe-96f1937c0d64/volumes"
Nov 03 20:31:10.086412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.086380    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"
Nov 03 20:31:10.112275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.112237    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:10.215811 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.215771    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56bf2d77-4688-4756-a9bb-01f7b346457c-kube-api-access-bs8cf\" (UniqueName: \"kubernetes.io/projected/56bf2d77-4688-4756-a9bb-01f7b346457c-kube-api-access-bs8cf\") pod \"56bf2d77-4688-4756-a9bb-01f7b346457c\" (UID: \"56bf2d77-4688-4756-a9bb-01f7b346457c\") "
Nov 03 20:31:10.216016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.215891    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-openshift-ca\") pod \"56bf2d77-4688-4756-a9bb-01f7b346457c\" (UID: \"56bf2d77-4688-4756-a9bb-01f7b346457c\") "
Nov 03 20:31:10.216139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.216124    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-xks-ca\") pod \"56bf2d77-4688-4756-a9bb-01f7b346457c\" (UID: \"56bf2d77-4688-4756-a9bb-01f7b346457c\") "
Nov 03 20:31:10.216277 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.216247    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-openshift-ca" pod "56bf2d77-4688-4756-a9bb-01f7b346457c" (UID: "56bf2d77-4688-4756-a9bb-01f7b346457c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:10.216852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.216830    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:10.218213 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.218190    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56bf2d77-4688-4756-a9bb-01f7b346457c-kube-api-access-bs8cf" pod "56bf2d77-4688-4756-a9bb-01f7b346457c" (UID: "56bf2d77-4688-4756-a9bb-01f7b346457c"). InnerVolumeSpecName "kube-api-access-bs8cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:10.218433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.218412    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-xks-ca" pod "56bf2d77-4688-4756-a9bb-01f7b346457c" (UID: "56bf2d77-4688-4756-a9bb-01f7b346457c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:10.242638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.242596    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:10.329527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.328223    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56bf2d77-4688-4756-a9bb-01f7b346457c-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:10.329527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.328260    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs8cf\" (UniqueName: \"kubernetes.io/projected/56bf2d77-4688-4756-a9bb-01f7b346457c-kube-api-access-bs8cf\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:10.404908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.404872    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:10.879440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.879407    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"
Nov 03 20:31:10.879695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.879413    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj" event={"ID":"56bf2d77-4688-4756-a9bb-01f7b346457c","Type":"ContainerDied","Data":"34280dce94d696ee4f70bef2bab56b2ca1364cad328abf298c0d0f2b9b10fa15"}
Nov 03 20:31:10.879695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.879586    5659 scope.go:122] "RemoveContainer" containerID="a176131294668e220f92409dfd9c63eef888b90da3613ca7134c81001e983c40"
Nov 03 20:31:10.912464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.912430    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"]
Nov 03 20:31:10.919834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:10.919803    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-tbfnj"]
Nov 03 20:31:11.046214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.046170    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:11.203467 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.203388    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:11.222333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.222300    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"]
Nov 03 20:31:11.223154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223139    5659 cpu_manager.go:414] "containerMap: removing container" podUID="56bf2d77-4688-4756-a9bb-01f7b346457c" containerName="revoke-keys"
Nov 03 20:31:11.223211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223156    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56bf2d77-4688-4756-a9bb-01f7b346457c" containerName="revoke-keys"
Nov 03 20:31:11.223211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223168    5659 cpu_manager.go:414] "containerMap: removing container" podUID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" containerName="revoke-keys"
Nov 03 20:31:11.223211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223174    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" containerName="revoke-keys"
Nov 03 20:31:11.223211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223187    5659 cpu_manager.go:414] "containerMap: removing container" podUID="56bf2d77-4688-4756-a9bb-01f7b346457c" containerName="revoke-keys"
Nov 03 20:31:11.223211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223192    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56bf2d77-4688-4756-a9bb-01f7b346457c" containerName="revoke-keys"
Nov 03 20:31:11.223211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223211    5659 cpu_manager.go:414] "containerMap: removing container" podUID="9e43233f-4023-428a-8cff-fffd9ab19d74" containerName="cleanup"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223215    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e43233f-4023-428a-8cff-fffd9ab19d74" containerName="cleanup"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223223    5659 cpu_manager.go:414] "containerMap: removing container" podUID="29399fbb-4180-4dcc-b2a0-4a18505e1fb9" containerName="cleanup"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223227    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29399fbb-4180-4dcc-b2a0-4a18505e1fb9" containerName="cleanup"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223251    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8f042faa-25c7-4ce0-9fb8-16fc7633f5d5" containerName="cleanup"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223256    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f042faa-25c7-4ce0-9fb8-16fc7633f5d5" containerName="cleanup"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223263    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64" containerName="revoke-keys"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223267    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64" containerName="revoke-keys"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223276    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64" containerName="revoke-keys"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223280    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64" containerName="revoke-keys"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223302    5659 cpu_manager.go:414] "containerMap: removing container" podUID="2bc004c1-979c-4f3e-ba19-fddcd5e58629" containerName="cleanup"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223306    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc004c1-979c-4f3e-ba19-fddcd5e58629" containerName="cleanup"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223318    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b88035b6-4c8a-4c03-b803-e897a2c86818" containerName="cleanup"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223323    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88035b6-4c8a-4c03-b803-e897a2c86818" containerName="cleanup"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223332    5659 cpu_manager.go:414] "containerMap: removing container" podUID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" containerName="revoke-keys"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223336    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" containerName="revoke-keys"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223344    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64" containerName="revoke-keys"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223349    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64" containerName="revoke-keys"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223358    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8f042faa-25c7-4ce0-9fb8-16fc7633f5d5" containerName="cleanup"
Nov 03 20:31:11.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223362    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f042faa-25c7-4ce0-9fb8-16fc7633f5d5" containerName="cleanup"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223473    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f042faa-25c7-4ce0-9fb8-16fc7633f5d5" containerName="cleanup"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223496    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64" containerName="revoke-keys"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223507    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64" containerName="revoke-keys"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223518    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="56bf2d77-4688-4756-a9bb-01f7b346457c" containerName="revoke-keys"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223533    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" containerName="revoke-keys"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223540    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e43233f-4023-428a-8cff-fffd9ab19d74" containerName="cleanup"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223552    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f1d55f-acb8-45e5-bffe-96f1937c0d64" containerName="revoke-keys"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223563    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f042faa-25c7-4ce0-9fb8-16fc7633f5d5" containerName="cleanup"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223576    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" containerName="revoke-keys"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223585    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="56bf2d77-4688-4756-a9bb-01f7b346457c" containerName="revoke-keys"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223592    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" containerName="revoke-keys"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223598    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c13e2c-aeb0-4eb2-90ed-78c91e40280e" containerName="cleanup"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223605    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e43233f-4023-428a-8cff-fffd9ab19d74" containerName="cleanup"
Nov 03 20:31:11.224312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.223620    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88035b6-4c8a-4c03-b803-e897a2c86818" containerName="cleanup"
Nov 03 20:31:11.228929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.228907    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"
Nov 03 20:31:11.256010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.252048    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"]
Nov 03 20:31:11.359574 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.359534    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkhkg\" (UniqueName: \"kubernetes.io/projected/824d4448-ed72-4179-b932-1ca4b7f98376-kube-api-access-mkhkg\") pod \"maas-api-revoke-sub-403cf98d-5lf7q\" (UID: \"824d4448-ed72-4179-b932-1ca4b7f98376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"
Nov 03 20:31:11.359829 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.359801    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-5lf7q\" (UID: \"824d4448-ed72-4179-b932-1ca4b7f98376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"
Nov 03 20:31:11.360078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.360057    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-5lf7q\" (UID: \"824d4448-ed72-4179-b932-1ca4b7f98376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"
Nov 03 20:31:11.461264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.461230    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-5lf7q\" (UID: \"824d4448-ed72-4179-b932-1ca4b7f98376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"
Nov 03 20:31:11.461502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.461323    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkhkg\" (UniqueName: \"kubernetes.io/projected/824d4448-ed72-4179-b932-1ca4b7f98376-kube-api-access-mkhkg\") pod \"maas-api-revoke-sub-403cf98d-5lf7q\" (UID: \"824d4448-ed72-4179-b932-1ca4b7f98376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"
Nov 03 20:31:11.461502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.461462    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-5lf7q\" (UID: \"824d4448-ed72-4179-b932-1ca4b7f98376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"
Nov 03 20:31:11.462091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.462068    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-5lf7q\" (UID: \"824d4448-ed72-4179-b932-1ca4b7f98376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"
Nov 03 20:31:11.464142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.464115    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-5lf7q\" (UID: \"824d4448-ed72-4179-b932-1ca4b7f98376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"
Nov 03 20:31:11.474842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.474802    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkhkg\" (UniqueName: \"kubernetes.io/projected/824d4448-ed72-4179-b932-1ca4b7f98376-kube-api-access-mkhkg\") pod \"maas-api-revoke-sub-403cf98d-5lf7q\" (UID: \"824d4448-ed72-4179-b932-1ca4b7f98376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"
Nov 03 20:31:11.557642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.557603    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"
Nov 03 20:31:11.729294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.729207    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:11.896316 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.896280    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"]
Nov 03 20:31:11.908341 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.908311    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q" event={"ID":"824d4448-ed72-4179-b932-1ca4b7f98376","Type":"ContainerStarted","Data":"8dc1d7610617dc405b6a68fa28dafaa09348b0a924138b13aa3b9cd4906c0425"}
Nov 03 20:31:11.986827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:11.986740    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56bf2d77-4688-4756-a9bb-01f7b346457c" path="/var/lib/kubelet/pods/56bf2d77-4688-4756-a9bb-01f7b346457c/volumes"
Nov 03 20:31:12.540568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.537126    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:12.728328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.728286    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"]
Nov 03 20:31:12.729601 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.729574    5659 cpu_manager.go:414] "containerMap: removing container" podUID="9e43233f-4023-428a-8cff-fffd9ab19d74" containerName="cleanup"
Nov 03 20:31:12.729601 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.729599    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e43233f-4023-428a-8cff-fffd9ab19d74" containerName="cleanup"
Nov 03 20:31:12.729823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.729682    5659 cpu_manager.go:414] "containerMap: removing container" podUID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" containerName="revoke-keys"
Nov 03 20:31:12.729823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.729690    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c1e43d-ebb8-4fe2-a1ec-ee5d7acd7c7b" containerName="revoke-keys"
Nov 03 20:31:12.729823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.729715    5659 cpu_manager.go:414] "containerMap: removing container" podUID="56bf2d77-4688-4756-a9bb-01f7b346457c" containerName="revoke-keys"
Nov 03 20:31:12.729823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.729723    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56bf2d77-4688-4756-a9bb-01f7b346457c" containerName="revoke-keys"
Nov 03 20:31:12.729976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.729890    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="56bf2d77-4688-4756-a9bb-01f7b346457c" containerName="revoke-keys"
Nov 03 20:31:12.736039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.736009    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"
Nov 03 20:31:12.743557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.743428    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"]
Nov 03 20:31:12.820018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.819914    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-4vdss\" (UID: \"6e214689-ceac-4756-9da8-586f6bbbac23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"
Nov 03 20:31:12.820600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.820222    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-4vdss\" (UID: \"6e214689-ceac-4756-9da8-586f6bbbac23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"
Nov 03 20:31:12.820745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.820651    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxb8s\" (UniqueName: \"kubernetes.io/projected/6e214689-ceac-4756-9da8-586f6bbbac23-kube-api-access-vxb8s\") pod \"maas-api-revoke-sub-5bac6f1e-4vdss\" (UID: \"6e214689-ceac-4756-9da8-586f6bbbac23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"
Nov 03 20:31:12.921139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.921102    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q" event={"ID":"824d4448-ed72-4179-b932-1ca4b7f98376","Type":"ContainerStarted","Data":"f668f329fa50ae9ff174626c4995b649b95f0e673a018b9063522302a8357f48"}
Nov 03 20:31:12.921938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.921911    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-4vdss\" (UID: \"6e214689-ceac-4756-9da8-586f6bbbac23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"
Nov 03 20:31:12.922080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.921962    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxb8s\" (UniqueName: \"kubernetes.io/projected/6e214689-ceac-4756-9da8-586f6bbbac23-kube-api-access-vxb8s\") pod \"maas-api-revoke-sub-5bac6f1e-4vdss\" (UID: \"6e214689-ceac-4756-9da8-586f6bbbac23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"
Nov 03 20:31:12.922080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.922052    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-4vdss\" (UID: \"6e214689-ceac-4756-9da8-586f6bbbac23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"
Nov 03 20:31:12.922621 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.922598    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-4vdss\" (UID: \"6e214689-ceac-4756-9da8-586f6bbbac23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"
Nov 03 20:31:12.924705 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.924677    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-4vdss\" (UID: \"6e214689-ceac-4756-9da8-586f6bbbac23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"
Nov 03 20:31:12.940102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.940065    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxb8s\" (UniqueName: \"kubernetes.io/projected/6e214689-ceac-4756-9da8-586f6bbbac23-kube-api-access-vxb8s\") pod \"maas-api-revoke-sub-5bac6f1e-4vdss\" (UID: \"6e214689-ceac-4756-9da8-586f6bbbac23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"
Nov 03 20:31:12.970959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:12.961705    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q" podStartSLOduration=1.9616808159999999 podStartE2EDuration="1.961680816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:12.945891009 +0000 UTC m=+3225.100793431" watchObservedRunningTime="2026-11-03 20:31:12.961680816 +0000 UTC m=+3225.116583240"
Nov 03 20:31:13.023969 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:13.023923    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:13.064759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:13.064719    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"
Nov 03 20:31:13.344780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:13.342295    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:13.540734 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:31:13.529217    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e214689_ceac_4756_9da8_586f6bbbac23.slice/crio-84e64bec851f794a88748a59e59939e6d86671b22c16ef346d2fa24192efe9a2 WatchSource:0}: Error finding container 84e64bec851f794a88748a59e59939e6d86671b22c16ef346d2fa24192efe9a2: Status 404 returned error can't find the container with id 84e64bec851f794a88748a59e59939e6d86671b22c16ef346d2fa24192efe9a2
Nov 03 20:31:13.581433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:13.575649    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"]
Nov 03 20:31:13.645402 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:13.645356    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb" podUID="98e6d798-5d9f-4ee9-94c5-7d7faf5caf59" containerName="payload-processing" containerID="cri-o://08e52b666c849af3e926daa4fe9db4a2a063c21af606eea0aa9c3e451967d0a3" gracePeriod=25
Nov 03 20:31:13.956094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:13.951099    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss" event={"ID":"6e214689-ceac-4756-9da8-586f6bbbac23","Type":"ContainerStarted","Data":"84e64bec851f794a88748a59e59939e6d86671b22c16ef346d2fa24192efe9a2"}
Nov 03 20:31:13.956094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:13.953701    5659 generic.go:356] "Generic (PLEG): container finished" podID="98e6d798-5d9f-4ee9-94c5-7d7faf5caf59" containerID="08e52b666c849af3e926daa4fe9db4a2a063c21af606eea0aa9c3e451967d0a3" exitCode=0
Nov 03 20:31:13.956094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:13.953780    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb" event={"ID":"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59","Type":"ContainerDied","Data":"08e52b666c849af3e926daa4fe9db4a2a063c21af606eea0aa9c3e451967d0a3"}
Nov 03 20:31:13.956094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:13.955953    5659 generic.go:356] "Generic (PLEG): container finished" podID="824d4448-ed72-4179-b932-1ca4b7f98376" containerID="f668f329fa50ae9ff174626c4995b649b95f0e673a018b9063522302a8357f48" exitCode=22
Nov 03 20:31:13.956094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:13.956043    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q" event={"ID":"824d4448-ed72-4179-b932-1ca4b7f98376","Type":"ContainerDied","Data":"f668f329fa50ae9ff174626c4995b649b95f0e673a018b9063522302a8357f48"}
Nov 03 20:31:13.956626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:13.956606    5659 scope.go:122] "RemoveContainer" containerID="f668f329fa50ae9ff174626c4995b649b95f0e673a018b9063522302a8357f48"
Nov 03 20:31:14.021775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:14.021740    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"
Nov 03 20:31:14.072019 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:14.069946    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:14.086508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:14.084420    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-plugins-config-volume\") pod \"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59\" (UID: \"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59\") "
Nov 03 20:31:14.086508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:14.084529    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-kube-api-access-6jlv9\" (UniqueName: \"kubernetes.io/projected/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-kube-api-access-6jlv9\") pod \"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59\" (UID: \"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59\") "
Nov 03 20:31:14.086508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:14.086060    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-plugins-config-volume" pod "98e6d798-5d9f-4ee9-94c5-7d7faf5caf59" (UID: "98e6d798-5d9f-4ee9-94c5-7d7faf5caf59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:14.113438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:14.093646    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-kube-api-access-6jlv9" pod "98e6d798-5d9f-4ee9-94c5-7d7faf5caf59" (UID: "98e6d798-5d9f-4ee9-94c5-7d7faf5caf59"). InnerVolumeSpecName "kube-api-access-6jlv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:14.187148 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:14.187115    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jlv9\" (UniqueName: \"kubernetes.io/projected/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-kube-api-access-6jlv9\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:14.187148 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:14.187147    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:14.909405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:14.909372    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:15.015595 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:15.015564    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"
Nov 03 20:31:15.015802 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:15.015566    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb" event={"ID":"98e6d798-5d9f-4ee9-94c5-7d7faf5caf59","Type":"ContainerDied","Data":"9a75b946dd7ae4d3be827f4f30cc9e362e7d32c9929a1cdb353e4d5cc4edb163"}
Nov 03 20:31:15.015802 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:15.015734    5659 scope.go:122] "RemoveContainer" containerID="08e52b666c849af3e926daa4fe9db4a2a063c21af606eea0aa9c3e451967d0a3"
Nov 03 20:31:15.018564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:15.018450    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q" event={"ID":"824d4448-ed72-4179-b932-1ca4b7f98376","Type":"ContainerStarted","Data":"c37340409e2c1541ef41c10fb2a82de01162cbf38224f962b6af5dace8447b96"}
Nov 03 20:31:15.022997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:15.022969    5659 generic.go:356] "Generic (PLEG): container finished" podID="6e214689-ceac-4756-9da8-586f6bbbac23" containerID="e8f9093e2c001fedbda4ed9b5615e403001fb491843729ee1c7bca10918ee833" exitCode=22
Nov 03 20:31:15.023124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:15.023089    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss" event={"ID":"6e214689-ceac-4756-9da8-586f6bbbac23","Type":"ContainerDied","Data":"e8f9093e2c001fedbda4ed9b5615e403001fb491843729ee1c7bca10918ee833"}
Nov 03 20:31:15.023515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:15.023456    5659 scope.go:122] "RemoveContainer" containerID="e8f9093e2c001fedbda4ed9b5615e403001fb491843729ee1c7bca10918ee833"
Nov 03 20:31:15.089016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:15.088937    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"]
Nov 03 20:31:15.094064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:15.094039    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7b70d-7f47c76cdb-tz8fb"]
Nov 03 20:31:15.221087 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:15.221051    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:31:15.841556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:15.841521    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:15.990551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:15.990501    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e6d798-5d9f-4ee9-94c5-7d7faf5caf59" path="/var/lib/kubelet/pods/98e6d798-5d9f-4ee9-94c5-7d7faf5caf59/volumes"
Nov 03 20:31:16.043990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:16.043950    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss" event={"ID":"6e214689-ceac-4756-9da8-586f6bbbac23","Type":"ContainerStarted","Data":"d474f76be99f937f0751056cc46fa3173a2543a9caaca1ed3c39123a6063fcd1"}
Nov 03 20:31:16.051124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:16.050989    5659 generic.go:356] "Generic (PLEG): container finished" podID="824d4448-ed72-4179-b932-1ca4b7f98376" containerID="c37340409e2c1541ef41c10fb2a82de01162cbf38224f962b6af5dace8447b96" exitCode=22
Nov 03 20:31:16.051124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:16.051095    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q" event={"ID":"824d4448-ed72-4179-b932-1ca4b7f98376","Type":"ContainerDied","Data":"c37340409e2c1541ef41c10fb2a82de01162cbf38224f962b6af5dace8447b96"}
Nov 03 20:31:16.051400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:16.051168    5659 scope.go:122] "RemoveContainer" containerID="f668f329fa50ae9ff174626c4995b649b95f0e673a018b9063522302a8357f48"
Nov 03 20:31:16.051797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:16.051771    5659 scope.go:122] "RemoveContainer" containerID="c37340409e2c1541ef41c10fb2a82de01162cbf38224f962b6af5dace8447b96"
Nov 03 20:31:16.052603 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:16.052323    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-5lf7q_odh-ai-gateway-infra(824d4448-ed72-4179-b932-1ca4b7f98376)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q" podUID="824d4448-ed72-4179-b932-1ca4b7f98376"
Nov 03 20:31:16.070230 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:16.070166    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss" podStartSLOduration=4.070148287 podStartE2EDuration="4.070148287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:16.064257464 +0000 UTC m=+3228.219159884" watchObservedRunningTime="2026-11-03 20:31:16.070148287 +0000 UTC m=+3228.225050710"
Nov 03 20:31:16.540095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:16.540052    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:17.067428 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:17.067400    5659 scope.go:122] "RemoveContainer" containerID="c37340409e2c1541ef41c10fb2a82de01162cbf38224f962b6af5dace8447b96"
Nov 03 20:31:17.068947 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:17.068911    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-5lf7q_odh-ai-gateway-infra(824d4448-ed72-4179-b932-1ca4b7f98376)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q" podUID="824d4448-ed72-4179-b932-1ca4b7f98376"
Nov 03 20:31:17.071085 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:17.071058    5659 generic.go:356] "Generic (PLEG): container finished" podID="6e214689-ceac-4756-9da8-586f6bbbac23" containerID="d474f76be99f937f0751056cc46fa3173a2543a9caaca1ed3c39123a6063fcd1" exitCode=22
Nov 03 20:31:17.071290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:17.071156    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss" event={"ID":"6e214689-ceac-4756-9da8-586f6bbbac23","Type":"ContainerDied","Data":"d474f76be99f937f0751056cc46fa3173a2543a9caaca1ed3c39123a6063fcd1"}
Nov 03 20:31:17.071290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:17.071251    5659 scope.go:122] "RemoveContainer" containerID="e8f9093e2c001fedbda4ed9b5615e403001fb491843729ee1c7bca10918ee833"
Nov 03 20:31:17.071962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:17.071943    5659 scope.go:122] "RemoveContainer" containerID="d474f76be99f937f0751056cc46fa3173a2543a9caaca1ed3c39123a6063fcd1"
Nov 03 20:31:17.072570 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:17.072538    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-4vdss_odh-ai-gateway-infra(6e214689-ceac-4756-9da8-586f6bbbac23)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss" podUID="6e214689-ceac-4756-9da8-586f6bbbac23"
Nov 03 20:31:17.264310 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:17.257618    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:17.606661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:17.606626    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:17.970227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:17.970197    5659 scope.go:122] "RemoveContainer" containerID="d27628f2851bd263cde76b03586aea89f5ad7de9a54afce019e7efa9b6aa5e45"
Nov 03 20:31:18.044707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:18.044674    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:18.145968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:18.145938    5659 scope.go:122] "RemoveContainer" containerID="d474f76be99f937f0751056cc46fa3173a2543a9caaca1ed3c39123a6063fcd1"
Nov 03 20:31:18.146467 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:18.146434    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-4vdss_odh-ai-gateway-infra(6e214689-ceac-4756-9da8-586f6bbbac23)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss" podUID="6e214689-ceac-4756-9da8-586f6bbbac23"
Nov 03 20:31:18.744562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:18.737735    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:18.961316 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:18.961283    5659 scope.go:122] "RemoveContainer" containerID="c9ac06ca0f2c26bb52caa86d4ccfa94541f56af86bcf9dad54af8efb7d8447cc"
Nov 03 20:31:19.159301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:19.159268    5659 generic.go:356] "Generic (PLEG): container finished" podID="0005549b-99f3-4bd9-b80d-3414d16e1389" containerID="2e0ba63ac2261088482bd7ec7935439dd710adb1cd93fc314128f5ed23be69cf" exitCode=22
Nov 03 20:31:19.159708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:19.159310    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b" event={"ID":"0005549b-99f3-4bd9-b80d-3414d16e1389","Type":"ContainerDied","Data":"2e0ba63ac2261088482bd7ec7935439dd710adb1cd93fc314128f5ed23be69cf"}
Nov 03 20:31:19.159708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:19.159390    5659 scope.go:122] "RemoveContainer" containerID="d27628f2851bd263cde76b03586aea89f5ad7de9a54afce019e7efa9b6aa5e45"
Nov 03 20:31:19.159951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:19.159930    5659 scope.go:122] "RemoveContainer" containerID="2e0ba63ac2261088482bd7ec7935439dd710adb1cd93fc314128f5ed23be69cf"
Nov 03 20:31:19.160471 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:19.160359    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-fmd6b_odh-ai-gateway-infra(0005549b-99f3-4bd9-b80d-3414d16e1389)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b" podUID="0005549b-99f3-4bd9-b80d-3414d16e1389"
Nov 03 20:31:19.558781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:19.558738    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:19.976574 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:19.976540    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"]
Nov 03 20:31:19.977537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:19.977510    5659 cpu_manager.go:414] "containerMap: removing container" podUID="98e6d798-5d9f-4ee9-94c5-7d7faf5caf59" containerName="payload-processing"
Nov 03 20:31:19.977537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:19.977536    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e6d798-5d9f-4ee9-94c5-7d7faf5caf59" containerName="payload-processing"
Nov 03 20:31:19.977736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:19.977682    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e6d798-5d9f-4ee9-94c5-7d7faf5caf59" containerName="payload-processing"
Nov 03 20:31:19.985991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:19.985960    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"]
Nov 03 20:31:19.986153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:19.986071    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"
Nov 03 20:31:20.126302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.126265    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-zj9s8\" (UID: \"4afae507-7a10-43c4-92ef-864f1ecf3bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"
Nov 03 20:31:20.126720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.126693    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhthr\" (UniqueName: \"kubernetes.io/projected/4afae507-7a10-43c4-92ef-864f1ecf3bde-kube-api-access-mhthr\") pod \"maas-api-revoke-sub-266d0609-zj9s8\" (UID: \"4afae507-7a10-43c4-92ef-864f1ecf3bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"
Nov 03 20:31:20.126895 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.126877    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-zj9s8\" (UID: \"4afae507-7a10-43c4-92ef-864f1ecf3bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"
Nov 03 20:31:20.129512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.129389    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:20.177584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.177546    5659 generic.go:356] "Generic (PLEG): container finished" podID="1c1247bb-a71b-4f74-8261-c56a31e13462" containerID="fc99378a783c3a847c53f3cc344af0371ade7ce8d0dfa610ecf4b83d91bc16f2" exitCode=7
Nov 03 20:31:20.178039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.177616    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw" event={"ID":"1c1247bb-a71b-4f74-8261-c56a31e13462","Type":"ContainerDied","Data":"fc99378a783c3a847c53f3cc344af0371ade7ce8d0dfa610ecf4b83d91bc16f2"}
Nov 03 20:31:20.178039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.177681    5659 scope.go:122] "RemoveContainer" containerID="c9ac06ca0f2c26bb52caa86d4ccfa94541f56af86bcf9dad54af8efb7d8447cc"
Nov 03 20:31:20.178154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.178140    5659 scope.go:122] "RemoveContainer" containerID="fc99378a783c3a847c53f3cc344af0371ade7ce8d0dfa610ecf4b83d91bc16f2"
Nov 03 20:31:20.178610 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:20.178573    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-knsvw_odh-ai-gateway-infra(1c1247bb-a71b-4f74-8261-c56a31e13462)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw" podUID="1c1247bb-a71b-4f74-8261-c56a31e13462"
Nov 03 20:31:20.198002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.197968    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"]
Nov 03 20:31:20.228841 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.228800    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhthr\" (UniqueName: \"kubernetes.io/projected/4afae507-7a10-43c4-92ef-864f1ecf3bde-kube-api-access-mhthr\") pod \"maas-api-revoke-sub-266d0609-zj9s8\" (UID: \"4afae507-7a10-43c4-92ef-864f1ecf3bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"
Nov 03 20:31:20.229021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.228928    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-zj9s8\" (UID: \"4afae507-7a10-43c4-92ef-864f1ecf3bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"
Nov 03 20:31:20.229021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.228951    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-zj9s8\" (UID: \"4afae507-7a10-43c4-92ef-864f1ecf3bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"
Nov 03 20:31:20.229917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.229883    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-zj9s8\" (UID: \"4afae507-7a10-43c4-92ef-864f1ecf3bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"
Nov 03 20:31:20.231592 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.231544    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-zj9s8\" (UID: \"4afae507-7a10-43c4-92ef-864f1ecf3bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"
Nov 03 20:31:20.244290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.244246    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhthr\" (UniqueName: \"kubernetes.io/projected/4afae507-7a10-43c4-92ef-864f1ecf3bde-kube-api-access-mhthr\") pod \"maas-api-revoke-sub-266d0609-zj9s8\" (UID: \"4afae507-7a10-43c4-92ef-864f1ecf3bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"
Nov 03 20:31:20.317072 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.317032    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"
Nov 03 20:31:20.390666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.370357    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:20.524529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.523400    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"
Nov 03 20:31:20.637074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.636214    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-openshift-ca\") pod \"0005549b-99f3-4bd9-b80d-3414d16e1389\" (UID: \"0005549b-99f3-4bd9-b80d-3414d16e1389\") "
Nov 03 20:31:20.637074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.636450    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-xks-ca\") pod \"0005549b-99f3-4bd9-b80d-3414d16e1389\" (UID: \"0005549b-99f3-4bd9-b80d-3414d16e1389\") "
Nov 03 20:31:20.637074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.636572    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0005549b-99f3-4bd9-b80d-3414d16e1389-kube-api-access-ls4bd\" (UniqueName: \"kubernetes.io/projected/0005549b-99f3-4bd9-b80d-3414d16e1389-kube-api-access-ls4bd\") pod \"0005549b-99f3-4bd9-b80d-3414d16e1389\" (UID: \"0005549b-99f3-4bd9-b80d-3414d16e1389\") "
Nov 03 20:31:20.641325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.640738    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0005549b-99f3-4bd9-b80d-3414d16e1389-kube-api-access-ls4bd" pod "0005549b-99f3-4bd9-b80d-3414d16e1389" (UID: "0005549b-99f3-4bd9-b80d-3414d16e1389"). InnerVolumeSpecName "kube-api-access-ls4bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:20.641325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.641063    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-openshift-ca" pod "0005549b-99f3-4bd9-b80d-3414d16e1389" (UID: "0005549b-99f3-4bd9-b80d-3414d16e1389"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:20.643558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.643193    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-xks-ca" pod "0005549b-99f3-4bd9-b80d-3414d16e1389" (UID: "0005549b-99f3-4bd9-b80d-3414d16e1389"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:20.741154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.741116    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls4bd\" (UniqueName: \"kubernetes.io/projected/0005549b-99f3-4bd9-b80d-3414d16e1389-kube-api-access-ls4bd\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:20.741154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.741150    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:20.741154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.741162    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0005549b-99f3-4bd9-b80d-3414d16e1389-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:20.766136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.766100    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"]
Nov 03 20:31:20.767695 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:31:20.767666    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afae507_7a10_43c4_92ef_864f1ecf3bde.slice/crio-8bf662f6018954176badcbaf60bcce16e5f22afb9a75d5717adde63973db975b WatchSource:0}: Error finding container 8bf662f6018954176badcbaf60bcce16e5f22afb9a75d5717adde63973db975b: Status 404 returned error can't find the container with id 8bf662f6018954176badcbaf60bcce16e5f22afb9a75d5717adde63973db975b
Nov 03 20:31:20.961804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:20.961772    5659 scope.go:122] "RemoveContainer" containerID="19d6c4413809f3cbee328e9f68146c18c1556a819983f4a265e40672faab6222"
Nov 03 20:31:21.149013 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:21.148982    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:21.192218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:21.192188    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"
Nov 03 20:31:21.192735 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:21.192189    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b" event={"ID":"0005549b-99f3-4bd9-b80d-3414d16e1389","Type":"ContainerDied","Data":"70c1859d1bf43df4baa583ae71302bce744f326386cac03a4d1764be71f32ffe"}
Nov 03 20:31:21.192735 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:21.192344    5659 scope.go:122] "RemoveContainer" containerID="2e0ba63ac2261088482bd7ec7935439dd710adb1cd93fc314128f5ed23be69cf"
Nov 03 20:31:21.196181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:21.196119    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8" event={"ID":"4afae507-7a10-43c4-92ef-864f1ecf3bde","Type":"ContainerStarted","Data":"8bf662f6018954176badcbaf60bcce16e5f22afb9a75d5717adde63973db975b"}
Nov 03 20:31:21.212859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:21.212833    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"]
Nov 03 20:31:21.240523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:21.229345    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"]
Nov 03 20:31:21.248552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:21.247718    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-fmd6b"]
Nov 03 20:31:22.055206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.055162    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0005549b-99f3-4bd9-b80d-3414d16e1389" path="/var/lib/kubelet/pods/0005549b-99f3-4bd9-b80d-3414d16e1389/volumes"
Nov 03 20:31:22.151974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.149355    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"
Nov 03 20:31:22.162441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.162349    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:22.215588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.215551    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"]
Nov 03 20:31:22.217459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217424    5659 cpu_manager.go:414] "containerMap: removing container" podUID="0005549b-99f3-4bd9-b80d-3414d16e1389" containerName="revoke-keys"
Nov 03 20:31:22.217459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217447    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0005549b-99f3-4bd9-b80d-3414d16e1389" containerName="revoke-keys"
Nov 03 20:31:22.217459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217461    5659 cpu_manager.go:414] "containerMap: removing container" podUID="1c1247bb-a71b-4f74-8261-c56a31e13462" containerName="revoke-keys"
Nov 03 20:31:22.217714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217467    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1247bb-a71b-4f74-8261-c56a31e13462" containerName="revoke-keys"
Nov 03 20:31:22.217714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217518    5659 cpu_manager.go:414] "containerMap: removing container" podUID="1c1247bb-a71b-4f74-8261-c56a31e13462" containerName="revoke-keys"
Nov 03 20:31:22.217714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217526    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1247bb-a71b-4f74-8261-c56a31e13462" containerName="revoke-keys"
Nov 03 20:31:22.217714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217573    5659 cpu_manager.go:414] "containerMap: removing container" podUID="0005549b-99f3-4bd9-b80d-3414d16e1389" containerName="revoke-keys"
Nov 03 20:31:22.217714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217578    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0005549b-99f3-4bd9-b80d-3414d16e1389" containerName="revoke-keys"
Nov 03 20:31:22.217994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217749    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="0005549b-99f3-4bd9-b80d-3414d16e1389" containerName="revoke-keys"
Nov 03 20:31:22.217994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217758    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1247bb-a71b-4f74-8261-c56a31e13462" containerName="revoke-keys"
Nov 03 20:31:22.217994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217767    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1247bb-a71b-4f74-8261-c56a31e13462" containerName="revoke-keys"
Nov 03 20:31:22.217994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217776    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="0005549b-99f3-4bd9-b80d-3414d16e1389" containerName="revoke-keys"
Nov 03 20:31:22.217994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217784    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="0005549b-99f3-4bd9-b80d-3414d16e1389" containerName="revoke-keys"
Nov 03 20:31:22.217994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.217793    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1247bb-a71b-4f74-8261-c56a31e13462" containerName="revoke-keys"
Nov 03 20:31:22.220799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.220765    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"
Nov 03 20:31:22.223237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.223207    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw" event={"ID":"1c1247bb-a71b-4f74-8261-c56a31e13462","Type":"ContainerDied","Data":"ed423095cc327dc77e2d26566bf6d83965bd7f129f221a216f9e4084b9963d55"}
Nov 03 20:31:22.223335 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.223270    5659 scope.go:122] "RemoveContainer" containerID="fc99378a783c3a847c53f3cc344af0371ade7ce8d0dfa610ecf4b83d91bc16f2"
Nov 03 20:31:22.223414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.223400    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"
Nov 03 20:31:22.233677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.232277    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"]
Nov 03 20:31:22.233905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.233876    5659 generic.go:356] "Generic (PLEG): container finished" podID="4afae507-7a10-43c4-92ef-864f1ecf3bde" containerID="f59512ecfd0b12c133b257593f578ad99b23b83f63c74827a5b6f7712f04c477" exitCode=22
Nov 03 20:31:22.234039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.234018    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8" event={"ID":"4afae507-7a10-43c4-92ef-864f1ecf3bde","Type":"ContainerDied","Data":"f59512ecfd0b12c133b257593f578ad99b23b83f63c74827a5b6f7712f04c477"}
Nov 03 20:31:22.234535 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.234512    5659 scope.go:122] "RemoveContainer" containerID="f59512ecfd0b12c133b257593f578ad99b23b83f63c74827a5b6f7712f04c477"
Nov 03 20:31:22.258000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.256374    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2" event={"ID":"3bf82ad8-facf-49ca-a847-65af5a5f4154","Type":"ContainerStarted","Data":"db23ec1f4833cbb41e9c5be64ff4987e6773718bb7e870b8c90a7cc8fbefb0aa"}
Nov 03 20:31:22.319546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.317249    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1247bb-a71b-4f74-8261-c56a31e13462-kube-api-access-4b74w\" (UniqueName: \"kubernetes.io/projected/1c1247bb-a71b-4f74-8261-c56a31e13462-kube-api-access-4b74w\") pod \"1c1247bb-a71b-4f74-8261-c56a31e13462\" (UID: \"1c1247bb-a71b-4f74-8261-c56a31e13462\") "
Nov 03 20:31:22.319546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.317661    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-openshift-ca\") pod \"1c1247bb-a71b-4f74-8261-c56a31e13462\" (UID: \"1c1247bb-a71b-4f74-8261-c56a31e13462\") "
Nov 03 20:31:22.319546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.317704    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-xks-ca\") pod \"1c1247bb-a71b-4f74-8261-c56a31e13462\" (UID: \"1c1247bb-a71b-4f74-8261-c56a31e13462\") "
Nov 03 20:31:22.319546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.318207    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvwfq\" (UniqueName: \"kubernetes.io/projected/6724b388-74ed-45a2-b283-f64245c0a06a-kube-api-access-dvwfq\") pod \"maas-api-revoke-sub-abad4031-4whvx\" (UID: \"6724b388-74ed-45a2-b283-f64245c0a06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"
Nov 03 20:31:22.319546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.318417    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-4whvx\" (UID: \"6724b388-74ed-45a2-b283-f64245c0a06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"
Nov 03 20:31:22.319546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.318757    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-4whvx\" (UID: \"6724b388-74ed-45a2-b283-f64245c0a06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"
Nov 03 20:31:22.407968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.374236    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-openshift-ca" pod "1c1247bb-a71b-4f74-8261-c56a31e13462" (UID: "1c1247bb-a71b-4f74-8261-c56a31e13462"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:22.407968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.379671    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1247bb-a71b-4f74-8261-c56a31e13462-kube-api-access-4b74w" pod "1c1247bb-a71b-4f74-8261-c56a31e13462" (UID: "1c1247bb-a71b-4f74-8261-c56a31e13462"). InnerVolumeSpecName "kube-api-access-4b74w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:22.407968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.402331    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-xks-ca" pod "1c1247bb-a71b-4f74-8261-c56a31e13462" (UID: "1c1247bb-a71b-4f74-8261-c56a31e13462"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:22.451927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.436187    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-4whvx\" (UID: \"6724b388-74ed-45a2-b283-f64245c0a06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"
Nov 03 20:31:22.451927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.436449    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-4whvx\" (UID: \"6724b388-74ed-45a2-b283-f64245c0a06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"
Nov 03 20:31:22.451927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.436588    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvwfq\" (UniqueName: \"kubernetes.io/projected/6724b388-74ed-45a2-b283-f64245c0a06a-kube-api-access-dvwfq\") pod \"maas-api-revoke-sub-abad4031-4whvx\" (UID: \"6724b388-74ed-45a2-b283-f64245c0a06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"
Nov 03 20:31:22.451927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.436731    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b74w\" (UniqueName: \"kubernetes.io/projected/1c1247bb-a71b-4f74-8261-c56a31e13462-kube-api-access-4b74w\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:22.451927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.436747    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:22.451927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.436763    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1247bb-a71b-4f74-8261-c56a31e13462-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:22.451927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.437785    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-4whvx\" (UID: \"6724b388-74ed-45a2-b283-f64245c0a06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"
Nov 03 20:31:22.451927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.447234    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-4whvx\" (UID: \"6724b388-74ed-45a2-b283-f64245c0a06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"
Nov 03 20:31:22.451927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.451804    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvwfq\" (UniqueName: \"kubernetes.io/projected/6724b388-74ed-45a2-b283-f64245c0a06a-kube-api-access-dvwfq\") pod \"maas-api-revoke-sub-abad4031-4whvx\" (UID: \"6724b388-74ed-45a2-b283-f64245c0a06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"
Nov 03 20:31:22.521867 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.521835    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"
Nov 03 20:31:22.570684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.570649    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"]
Nov 03 20:31:22.576311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.576280    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-knsvw"]
Nov 03 20:31:22.831567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:22.831531    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"]
Nov 03 20:31:22.844891 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:31:22.844859    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6724b388_74ed_45a2_b283_f64245c0a06a.slice/crio-7df7a5acbcfb1bfb5c12428ff76741c62962d87352914203d00fb2e2dffda1d6 WatchSource:0}: Error finding container 7df7a5acbcfb1bfb5c12428ff76741c62962d87352914203d00fb2e2dffda1d6: Status 404 returned error can't find the container with id 7df7a5acbcfb1bfb5c12428ff76741c62962d87352914203d00fb2e2dffda1d6
Nov 03 20:31:23.294568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:23.293780    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8" event={"ID":"4afae507-7a10-43c4-92ef-864f1ecf3bde","Type":"ContainerStarted","Data":"7b4c6687c05f07a8b1d9aa25a8763a8719dd750d7709cf6465a949c641ace113"}
Nov 03 20:31:23.298821 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:23.298789    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx" event={"ID":"6724b388-74ed-45a2-b283-f64245c0a06a","Type":"ContainerStarted","Data":"7df7a5acbcfb1bfb5c12428ff76741c62962d87352914203d00fb2e2dffda1d6"}
Nov 03 20:31:23.304559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:23.304154    5659 generic.go:356] "Generic (PLEG): container finished" podID="3bf82ad8-facf-49ca-a847-65af5a5f4154" containerID="db23ec1f4833cbb41e9c5be64ff4987e6773718bb7e870b8c90a7cc8fbefb0aa" exitCode=22
Nov 03 20:31:23.304559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:23.304228    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2" event={"ID":"3bf82ad8-facf-49ca-a847-65af5a5f4154","Type":"ContainerDied","Data":"db23ec1f4833cbb41e9c5be64ff4987e6773718bb7e870b8c90a7cc8fbefb0aa"}
Nov 03 20:31:23.304559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:23.304268    5659 scope.go:122] "RemoveContainer" containerID="19d6c4413809f3cbee328e9f68146c18c1556a819983f4a265e40672faab6222"
Nov 03 20:31:23.304814 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:23.304584    5659 scope.go:122] "RemoveContainer" containerID="db23ec1f4833cbb41e9c5be64ff4987e6773718bb7e870b8c90a7cc8fbefb0aa"
Nov 03 20:31:23.304883 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:23.304840    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-6frs2_odh-ai-gateway-infra(3bf82ad8-facf-49ca-a847-65af5a5f4154)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2" podUID="3bf82ad8-facf-49ca-a847-65af5a5f4154"
Nov 03 20:31:23.316716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:23.316261    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8" podStartSLOduration=4.31624659 podStartE2EDuration="4.31624659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:23.316194673 +0000 UTC m=+3235.471097094" watchObservedRunningTime="2026-11-03 20:31:23.31624659 +0000 UTC m=+3235.471149011"
Nov 03 20:31:23.367296 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:23.367202    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:23.451548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:23.445452    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"]
Nov 03 20:31:23.994373 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:23.994329    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c1247bb-a71b-4f74-8261-c56a31e13462" path="/var/lib/kubelet/pods/1c1247bb-a71b-4f74-8261-c56a31e13462/volumes"
Nov 03 20:31:24.316522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.316456    5659 generic.go:356] "Generic (PLEG): container finished" podID="4afae507-7a10-43c4-92ef-864f1ecf3bde" containerID="7b4c6687c05f07a8b1d9aa25a8763a8719dd750d7709cf6465a949c641ace113" exitCode=22
Nov 03 20:31:24.316986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.316521    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8" event={"ID":"4afae507-7a10-43c4-92ef-864f1ecf3bde","Type":"ContainerDied","Data":"7b4c6687c05f07a8b1d9aa25a8763a8719dd750d7709cf6465a949c641ace113"}
Nov 03 20:31:24.316986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.316585    5659 scope.go:122] "RemoveContainer" containerID="f59512ecfd0b12c133b257593f578ad99b23b83f63c74827a5b6f7712f04c477"
Nov 03 20:31:24.316986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.316949    5659 scope.go:122] "RemoveContainer" containerID="7b4c6687c05f07a8b1d9aa25a8763a8719dd750d7709cf6465a949c641ace113"
Nov 03 20:31:24.317402 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:24.317367    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-zj9s8_odh-ai-gateway-infra(4afae507-7a10-43c4-92ef-864f1ecf3bde)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8" podUID="4afae507-7a10-43c4-92ef-864f1ecf3bde"
Nov 03 20:31:24.318641 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.318617    5659 generic.go:356] "Generic (PLEG): container finished" podID="6724b388-74ed-45a2-b283-f64245c0a06a" containerID="8241a397565f380b2f98768003dc91bc12e4661a07ae558254796e88ddce33e4" exitCode=22
Nov 03 20:31:24.318759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.318700    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx" event={"ID":"6724b388-74ed-45a2-b283-f64245c0a06a","Type":"ContainerDied","Data":"8241a397565f380b2f98768003dc91bc12e4661a07ae558254796e88ddce33e4"}
Nov 03 20:31:24.318995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.318981    5659 scope.go:122] "RemoveContainer" containerID="8241a397565f380b2f98768003dc91bc12e4661a07ae558254796e88ddce33e4"
Nov 03 20:31:24.632507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.632355    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:24.815528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.813576    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"
Nov 03 20:31:24.870178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.868871    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf82ad8-facf-49ca-a847-65af5a5f4154-kube-api-access-t5jk2\" (UniqueName: \"kubernetes.io/projected/3bf82ad8-facf-49ca-a847-65af5a5f4154-kube-api-access-t5jk2\") pod \"3bf82ad8-facf-49ca-a847-65af5a5f4154\" (UID: \"3bf82ad8-facf-49ca-a847-65af5a5f4154\") "
Nov 03 20:31:24.870178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.869044    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-openshift-ca\") pod \"3bf82ad8-facf-49ca-a847-65af5a5f4154\" (UID: \"3bf82ad8-facf-49ca-a847-65af5a5f4154\") "
Nov 03 20:31:24.870178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.869435    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-xks-ca\") pod \"3bf82ad8-facf-49ca-a847-65af5a5f4154\" (UID: \"3bf82ad8-facf-49ca-a847-65af5a5f4154\") "
Nov 03 20:31:24.880517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.880450    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-openshift-ca" pod "3bf82ad8-facf-49ca-a847-65af5a5f4154" (UID: "3bf82ad8-facf-49ca-a847-65af5a5f4154"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:24.933262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.933179    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf82ad8-facf-49ca-a847-65af5a5f4154-kube-api-access-t5jk2" pod "3bf82ad8-facf-49ca-a847-65af5a5f4154" (UID: "3bf82ad8-facf-49ca-a847-65af5a5f4154"). InnerVolumeSpecName "kube-api-access-t5jk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:24.933984 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.933942    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-xks-ca" pod "3bf82ad8-facf-49ca-a847-65af5a5f4154" (UID: "3bf82ad8-facf-49ca-a847-65af5a5f4154"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:24.974328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.974282    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:24.974328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.974328    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5jk2\" (UniqueName: \"kubernetes.io/projected/3bf82ad8-facf-49ca-a847-65af5a5f4154-kube-api-access-t5jk2\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:24.974620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:24.974347    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf82ad8-facf-49ca-a847-65af5a5f4154-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:25.332559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:25.331733    5659 scope.go:122] "RemoveContainer" containerID="7b4c6687c05f07a8b1d9aa25a8763a8719dd750d7709cf6465a949c641ace113"
Nov 03 20:31:25.333634 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:25.333598    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-zj9s8_odh-ai-gateway-infra(4afae507-7a10-43c4-92ef-864f1ecf3bde)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8" podUID="4afae507-7a10-43c4-92ef-864f1ecf3bde"
Nov 03 20:31:25.333950 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:25.333918    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx" event={"ID":"6724b388-74ed-45a2-b283-f64245c0a06a","Type":"ContainerStarted","Data":"fd613968f1fcebaa41b0cf19e87f877588d81d5f963bbc5eb609dad4b0ca5dd0"}
Nov 03 20:31:25.336939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:25.336913    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"
Nov 03 20:31:25.337047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:25.336932    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2" event={"ID":"3bf82ad8-facf-49ca-a847-65af5a5f4154","Type":"ContainerDied","Data":"afb7c10e212005eb4cf6f968862a1d4da762a0f5b5476cc1aa519700787f0926"}
Nov 03 20:31:25.337047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:25.336998    5659 scope.go:122] "RemoveContainer" containerID="db23ec1f4833cbb41e9c5be64ff4987e6773718bb7e870b8c90a7cc8fbefb0aa"
Nov 03 20:31:25.371838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:25.371785    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx" podStartSLOduration=3.371769787 podStartE2EDuration="3.371769787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:25.369853229 +0000 UTC m=+3237.524755651" watchObservedRunningTime="2026-11-03 20:31:25.371769787 +0000 UTC m=+3237.526672614"
Nov 03 20:31:25.386592 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:25.386559    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"]
Nov 03 20:31:25.395597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:25.395567    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-6frs2"]
Nov 03 20:31:25.599657 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:25.599009    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:25.973388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:25.973351    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf82ad8-facf-49ca-a847-65af5a5f4154" path="/var/lib/kubelet/pods/3bf82ad8-facf-49ca-a847-65af5a5f4154/volumes"
Nov 03 20:31:26.330080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:26.329991    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:26.427564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:26.427467    5659 generic.go:356] "Generic (PLEG): container finished" podID="6724b388-74ed-45a2-b283-f64245c0a06a" containerID="fd613968f1fcebaa41b0cf19e87f877588d81d5f963bbc5eb609dad4b0ca5dd0" exitCode=22
Nov 03 20:31:26.428106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:26.428087    5659 scope.go:122] "RemoveContainer" containerID="fd613968f1fcebaa41b0cf19e87f877588d81d5f963bbc5eb609dad4b0ca5dd0"
Nov 03 20:31:26.428464 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:26.428436    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-4whvx_odh-ai-gateway-infra(6724b388-74ed-45a2-b283-f64245c0a06a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx" podUID="6724b388-74ed-45a2-b283-f64245c0a06a"
Nov 03 20:31:26.428761 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:26.428714    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx" event={"ID":"6724b388-74ed-45a2-b283-f64245c0a06a","Type":"ContainerDied","Data":"fd613968f1fcebaa41b0cf19e87f877588d81d5f963bbc5eb609dad4b0ca5dd0"}
Nov 03 20:31:26.428838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:26.428784    5659 scope.go:122] "RemoveContainer" containerID="8241a397565f380b2f98768003dc91bc12e4661a07ae558254796e88ddce33e4"
Nov 03 20:31:26.962365 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:26.962334    5659 scope.go:122] "RemoveContainer" containerID="c37340409e2c1541ef41c10fb2a82de01162cbf38224f962b6af5dace8447b96"
Nov 03 20:31:27.158661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:27.158584    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:27.410945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:27.410912    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:27.445859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:27.445777    5659 scope.go:122] "RemoveContainer" containerID="fd613968f1fcebaa41b0cf19e87f877588d81d5f963bbc5eb609dad4b0ca5dd0"
Nov 03 20:31:27.446368 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:27.446171    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-4whvx_odh-ai-gateway-infra(6724b388-74ed-45a2-b283-f64245c0a06a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx" podUID="6724b388-74ed-45a2-b283-f64245c0a06a"
Nov 03 20:31:27.448546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:27.448472    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q" event={"ID":"824d4448-ed72-4179-b932-1ca4b7f98376","Type":"ContainerStarted","Data":"1cff804032848f3d0710472e14de1148f5bc560a8c8470c7fc9d9756d73338a5"}
Nov 03 20:31:28.013648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.013581    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:28.461009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.460973    5659 generic.go:356] "Generic (PLEG): container finished" podID="824d4448-ed72-4179-b932-1ca4b7f98376" containerID="1cff804032848f3d0710472e14de1148f5bc560a8c8470c7fc9d9756d73338a5" exitCode=22
Nov 03 20:31:28.461532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.461020    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q" event={"ID":"824d4448-ed72-4179-b932-1ca4b7f98376","Type":"ContainerDied","Data":"1cff804032848f3d0710472e14de1148f5bc560a8c8470c7fc9d9756d73338a5"}
Nov 03 20:31:28.461532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.461082    5659 scope.go:122] "RemoveContainer" containerID="c37340409e2c1541ef41c10fb2a82de01162cbf38224f962b6af5dace8447b96"
Nov 03 20:31:28.461644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.461628    5659 scope.go:122] "RemoveContainer" containerID="1cff804032848f3d0710472e14de1148f5bc560a8c8470c7fc9d9756d73338a5"
Nov 03 20:31:28.462369 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:28.462134    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-5lf7q_odh-ai-gateway-infra(824d4448-ed72-4179-b932-1ca4b7f98376)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q" podUID="824d4448-ed72-4179-b932-1ca4b7f98376"
Nov 03 20:31:28.497704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.497674    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"]
Nov 03 20:31:28.730597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.730511    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"]
Nov 03 20:31:28.731400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.731384    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3bf82ad8-facf-49ca-a847-65af5a5f4154" containerName="revoke-keys"
Nov 03 20:31:28.731447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.731400    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf82ad8-facf-49ca-a847-65af5a5f4154" containerName="revoke-keys"
Nov 03 20:31:28.731447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.731426    5659 cpu_manager.go:414] "containerMap: removing container" podUID="0005549b-99f3-4bd9-b80d-3414d16e1389" containerName="revoke-keys"
Nov 03 20:31:28.731447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.731432    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0005549b-99f3-4bd9-b80d-3414d16e1389" containerName="revoke-keys"
Nov 03 20:31:28.731447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.731444    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3bf82ad8-facf-49ca-a847-65af5a5f4154" containerName="revoke-keys"
Nov 03 20:31:28.731630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.731450    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf82ad8-facf-49ca-a847-65af5a5f4154" containerName="revoke-keys"
Nov 03 20:31:28.731630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.731462    5659 cpu_manager.go:414] "containerMap: removing container" podUID="1c1247bb-a71b-4f74-8261-c56a31e13462" containerName="revoke-keys"
Nov 03 20:31:28.731630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.731467    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1247bb-a71b-4f74-8261-c56a31e13462" containerName="revoke-keys"
Nov 03 20:31:28.731630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.731475    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3bf82ad8-facf-49ca-a847-65af5a5f4154" containerName="revoke-keys"
Nov 03 20:31:28.731630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.731554    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf82ad8-facf-49ca-a847-65af5a5f4154" containerName="revoke-keys"
Nov 03 20:31:28.731797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.731783    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf82ad8-facf-49ca-a847-65af5a5f4154" containerName="revoke-keys"
Nov 03 20:31:28.731831 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.731823    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf82ad8-facf-49ca-a847-65af5a5f4154" containerName="revoke-keys"
Nov 03 20:31:28.738059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.738034    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"
Nov 03 20:31:28.750653 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.745751    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"]
Nov 03 20:31:28.836875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.836845    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:28.876522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.876073    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-n5gzh\" (UID: \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"
Nov 03 20:31:28.916566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.914010    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-n5gzh\" (UID: \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"
Nov 03 20:31:28.916894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:28.916860    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmq52\" (UniqueName: \"kubernetes.io/projected/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-kube-api-access-tmq52\") pod \"maas-api-revoke-sub-fb827799-n5gzh\" (UID: \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"
Nov 03 20:31:29.031712 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.031604    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-n5gzh\" (UID: \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"
Nov 03 20:31:29.031902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.031767    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-n5gzh\" (UID: \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"
Nov 03 20:31:29.031902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.031821    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmq52\" (UniqueName: \"kubernetes.io/projected/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-kube-api-access-tmq52\") pod \"maas-api-revoke-sub-fb827799-n5gzh\" (UID: \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"
Nov 03 20:31:29.032501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.032459    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-n5gzh\" (UID: \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"
Nov 03 20:31:29.034301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.034274    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-n5gzh\" (UID: \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"
Nov 03 20:31:29.043830 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.043792    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmq52\" (UniqueName: \"kubernetes.io/projected/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-kube-api-access-tmq52\") pod \"maas-api-revoke-sub-fb827799-n5gzh\" (UID: \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"
Nov 03 20:31:29.070018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.069987    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"
Nov 03 20:31:29.357522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.357474    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"]
Nov 03 20:31:29.358528 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:31:29.358449    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63ae334_c34a_4f9f_b7c8_3c82a0cb84b0.slice/crio-b1d49eddf548687b560c829fabdca390a44b8d4cb5799419888283f2cf95e34b WatchSource:0}: Error finding container b1d49eddf548687b560c829fabdca390a44b8d4cb5799419888283f2cf95e34b: Status 404 returned error can't find the container with id b1d49eddf548687b560c829fabdca390a44b8d4cb5799419888283f2cf95e34b
Nov 03 20:31:29.474879 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.474841    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh" event={"ID":"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0","Type":"ContainerStarted","Data":"b1d49eddf548687b560c829fabdca390a44b8d4cb5799419888283f2cf95e34b"}
Nov 03 20:31:29.589627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.581458    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:29.751831 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.751802    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"
Nov 03 20:31:29.851279 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.851238    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824d4448-ed72-4179-b932-1ca4b7f98376-kube-api-access-mkhkg\" (UniqueName: \"kubernetes.io/projected/824d4448-ed72-4179-b932-1ca4b7f98376-kube-api-access-mkhkg\") pod \"824d4448-ed72-4179-b932-1ca4b7f98376\" (UID: \"824d4448-ed72-4179-b932-1ca4b7f98376\") "
Nov 03 20:31:29.851531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.851309    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-openshift-ca\") pod \"824d4448-ed72-4179-b932-1ca4b7f98376\" (UID: \"824d4448-ed72-4179-b932-1ca4b7f98376\") "
Nov 03 20:31:29.851531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.851477    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-xks-ca\") pod \"824d4448-ed72-4179-b932-1ca4b7f98376\" (UID: \"824d4448-ed72-4179-b932-1ca4b7f98376\") "
Nov 03 20:31:29.851941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.851910    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-openshift-ca" pod "824d4448-ed72-4179-b932-1ca4b7f98376" (UID: "824d4448-ed72-4179-b932-1ca4b7f98376"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:29.852586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.852556    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:29.853761 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.853734    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824d4448-ed72-4179-b932-1ca4b7f98376-kube-api-access-mkhkg" pod "824d4448-ed72-4179-b932-1ca4b7f98376" (UID: "824d4448-ed72-4179-b932-1ca4b7f98376"). InnerVolumeSpecName "kube-api-access-mkhkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:29.853849 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.853770    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-xks-ca" pod "824d4448-ed72-4179-b932-1ca4b7f98376" (UID: "824d4448-ed72-4179-b932-1ca4b7f98376"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:29.895233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.895175    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:29.954243 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.954203    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkhkg\" (UniqueName: \"kubernetes.io/projected/824d4448-ed72-4179-b932-1ca4b7f98376-kube-api-access-mkhkg\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:29.954243 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:29.954238    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824d4448-ed72-4179-b932-1ca4b7f98376-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:30.028688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:30.028654    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:30.241822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:30.241784    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:30.492465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:30.492385    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"
Nov 03 20:31:30.492465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:30.492416    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q" event={"ID":"824d4448-ed72-4179-b932-1ca4b7f98376","Type":"ContainerDied","Data":"8dc1d7610617dc405b6a68fa28dafaa09348b0a924138b13aa3b9cd4906c0425"}
Nov 03 20:31:30.493062 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:30.492500    5659 scope.go:122] "RemoveContainer" containerID="1cff804032848f3d0710472e14de1148f5bc560a8c8470c7fc9d9756d73338a5"
Nov 03 20:31:30.494804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:30.494775    5659 generic.go:356] "Generic (PLEG): container finished" podID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" containerID="8e8dbee1aaa05a7097eddf54ebceb052ea44deec5a0d78ceec2c8fa26d1bf4e7" exitCode=7
Nov 03 20:31:30.494985 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:30.494965    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh" event={"ID":"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0","Type":"ContainerDied","Data":"8e8dbee1aaa05a7097eddf54ebceb052ea44deec5a0d78ceec2c8fa26d1bf4e7"}
Nov 03 20:31:30.495367 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:30.495348    5659 scope.go:122] "RemoveContainer" containerID="8e8dbee1aaa05a7097eddf54ebceb052ea44deec5a0d78ceec2c8fa26d1bf4e7"
Nov 03 20:31:30.516521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:30.516456    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"]
Nov 03 20:31:30.522755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:30.522729    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5lf7q"]
Nov 03 20:31:30.962352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:30.962313    5659 scope.go:122] "RemoveContainer" containerID="d474f76be99f937f0751056cc46fa3173a2543a9caaca1ed3c39123a6063fcd1"
Nov 03 20:31:31.028327 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.028285    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:31.254438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.254398    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"]
Nov 03 20:31:31.255293 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.255274    5659 cpu_manager.go:414] "containerMap: removing container" podUID="824d4448-ed72-4179-b932-1ca4b7f98376" containerName="revoke-keys"
Nov 03 20:31:31.255293 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.255291    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824d4448-ed72-4179-b932-1ca4b7f98376" containerName="revoke-keys"
Nov 03 20:31:31.255461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.255328    5659 cpu_manager.go:414] "containerMap: removing container" podUID="824d4448-ed72-4179-b932-1ca4b7f98376" containerName="revoke-keys"
Nov 03 20:31:31.255461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.255333    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824d4448-ed72-4179-b932-1ca4b7f98376" containerName="revoke-keys"
Nov 03 20:31:31.255461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.255347    5659 cpu_manager.go:414] "containerMap: removing container" podUID="824d4448-ed72-4179-b932-1ca4b7f98376" containerName="revoke-keys"
Nov 03 20:31:31.255461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.255353    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824d4448-ed72-4179-b932-1ca4b7f98376" containerName="revoke-keys"
Nov 03 20:31:31.255696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.255465    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="824d4448-ed72-4179-b932-1ca4b7f98376" containerName="revoke-keys"
Nov 03 20:31:31.255696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.255475    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf82ad8-facf-49ca-a847-65af5a5f4154" containerName="revoke-keys"
Nov 03 20:31:31.255696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.255499    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="824d4448-ed72-4179-b932-1ca4b7f98376" containerName="revoke-keys"
Nov 03 20:31:31.255696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.255508    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="824d4448-ed72-4179-b932-1ca4b7f98376" containerName="revoke-keys"
Nov 03 20:31:31.259418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.259396    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"
Nov 03 20:31:31.282605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.282571    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"]
Nov 03 20:31:31.376280 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.376247    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:31.402384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.402312    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-jvbjk\" (UID: \"025ae598-4df8-4574-a9b2-677c19a15fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"
Nov 03 20:31:31.402529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.402498    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-jvbjk\" (UID: \"025ae598-4df8-4574-a9b2-677c19a15fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"
Nov 03 20:31:31.402749 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.402720    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks94q\" (UniqueName: \"kubernetes.io/projected/025ae598-4df8-4574-a9b2-677c19a15fb7-kube-api-access-ks94q\") pod \"maas-api-revoke-sub-403cf98d-jvbjk\" (UID: \"025ae598-4df8-4574-a9b2-677c19a15fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"
Nov 03 20:31:31.504145 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.504108    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-jvbjk\" (UID: \"025ae598-4df8-4574-a9b2-677c19a15fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"
Nov 03 20:31:31.504686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.504222    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks94q\" (UniqueName: \"kubernetes.io/projected/025ae598-4df8-4574-a9b2-677c19a15fb7-kube-api-access-ks94q\") pod \"maas-api-revoke-sub-403cf98d-jvbjk\" (UID: \"025ae598-4df8-4574-a9b2-677c19a15fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"
Nov 03 20:31:31.504686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.504330    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-jvbjk\" (UID: \"025ae598-4df8-4574-a9b2-677c19a15fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"
Nov 03 20:31:31.505048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.505026    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-jvbjk\" (UID: \"025ae598-4df8-4574-a9b2-677c19a15fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"
Nov 03 20:31:31.506832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.506802    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss" event={"ID":"6e214689-ceac-4756-9da8-586f6bbbac23","Type":"ContainerStarted","Data":"74c51b9a550d32444b6fecae53c37d3c49d101d8012a892b95e0f2253439658f"}
Nov 03 20:31:31.506949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.506919    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-jvbjk\" (UID: \"025ae598-4df8-4574-a9b2-677c19a15fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"
Nov 03 20:31:31.508450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.508426    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:31.508974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.508952    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh" event={"ID":"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0","Type":"ContainerStarted","Data":"1bc66b4ad22257c3a190b44674829b850bf4d2880a03d37539b3bcd2e1c58c61"}
Nov 03 20:31:31.516738 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.516719    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks94q\" (UniqueName: \"kubernetes.io/projected/025ae598-4df8-4574-a9b2-677c19a15fb7-kube-api-access-ks94q\") pod \"maas-api-revoke-sub-403cf98d-jvbjk\" (UID: \"025ae598-4df8-4574-a9b2-677c19a15fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"
Nov 03 20:31:31.548263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.548190    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh" podStartSLOduration=3.548170956 podStartE2EDuration="3.548170956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:31.543794451 +0000 UTC m=+3243.698696872" watchObservedRunningTime="2026-11-03 20:31:31.548170956 +0000 UTC m=+3243.703073376"
Nov 03 20:31:31.584075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.584044    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"
Nov 03 20:31:31.861914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.861883    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:31.986441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:31.986402    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824d4448-ed72-4179-b932-1ca4b7f98376" path="/var/lib/kubelet/pods/824d4448-ed72-4179-b932-1ca4b7f98376/volumes"
Nov 03 20:31:32.010566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.002855    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"]
Nov 03 20:31:32.523169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.523079    5659 generic.go:356] "Generic (PLEG): container finished" podID="6e214689-ceac-4756-9da8-586f6bbbac23" containerID="74c51b9a550d32444b6fecae53c37d3c49d101d8012a892b95e0f2253439658f" exitCode=22
Nov 03 20:31:32.523169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.523127    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss" event={"ID":"6e214689-ceac-4756-9da8-586f6bbbac23","Type":"ContainerDied","Data":"74c51b9a550d32444b6fecae53c37d3c49d101d8012a892b95e0f2253439658f"}
Nov 03 20:31:32.523716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.523192    5659 scope.go:122] "RemoveContainer" containerID="d474f76be99f937f0751056cc46fa3173a2543a9caaca1ed3c39123a6063fcd1"
Nov 03 20:31:32.523716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.523704    5659 scope.go:122] "RemoveContainer" containerID="74c51b9a550d32444b6fecae53c37d3c49d101d8012a892b95e0f2253439658f"
Nov 03 20:31:32.524207 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:32.524179    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-4vdss_odh-ai-gateway-infra(6e214689-ceac-4756-9da8-586f6bbbac23)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss" podUID="6e214689-ceac-4756-9da8-586f6bbbac23"
Nov 03 20:31:32.525733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.525707    5659 generic.go:356] "Generic (PLEG): container finished" podID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" containerID="1bc66b4ad22257c3a190b44674829b850bf4d2880a03d37539b3bcd2e1c58c61" exitCode=7
Nov 03 20:31:32.525845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.525830    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh" event={"ID":"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0","Type":"ContainerDied","Data":"1bc66b4ad22257c3a190b44674829b850bf4d2880a03d37539b3bcd2e1c58c61"}
Nov 03 20:31:32.526180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.526163    5659 scope.go:122] "RemoveContainer" containerID="1bc66b4ad22257c3a190b44674829b850bf4d2880a03d37539b3bcd2e1c58c61"
Nov 03 20:31:32.526687 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:32.526653    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-n5gzh_odh-ai-gateway-infra(d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh" podUID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0"
Nov 03 20:31:32.527686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.527663    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk" event={"ID":"025ae598-4df8-4574-a9b2-677c19a15fb7","Type":"ContainerStarted","Data":"0aa0f7d1dee3205ab95652f03353871041be1ec3f7990f70e392f15c9e576c2b"}
Nov 03 20:31:32.527793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.527703    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk" event={"ID":"025ae598-4df8-4574-a9b2-677c19a15fb7","Type":"ContainerStarted","Data":"83dd1fc66e7874c37d745d9dbec9b37b628a4ee7259cc6ef4efb7e0960f7882c"}
Nov 03 20:31:32.541198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.541149    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"]
Nov 03 20:31:32.542096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.542063    5659 scope.go:122] "RemoveContainer" containerID="8e8dbee1aaa05a7097eddf54ebceb052ea44deec5a0d78ceec2c8fa26d1bf4e7"
Nov 03 20:31:32.585016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.584964    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk" podStartSLOduration=1.584948824 podStartE2EDuration="1.584948824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:32.582032429 +0000 UTC m=+3244.736934848" watchObservedRunningTime="2026-11-03 20:31:32.584948824 +0000 UTC m=+3244.739851242"
Nov 03 20:31:32.641569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.641466    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"]
Nov 03 20:31:32.649787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.648926    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"
Nov 03 20:31:32.654342 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.654313    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"]
Nov 03 20:31:32.824324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.824197    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:32.826117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.826090    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvfp\" (UniqueName: \"kubernetes.io/projected/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-kube-api-access-6bvfp\") pod \"maas-api-revoke-sub-f14db2bd-jnzr2\" (UID: \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"
Nov 03 20:31:32.826520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.826464    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-jnzr2\" (UID: \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"
Nov 03 20:31:32.826816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.826799    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-jnzr2\" (UID: \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"
Nov 03 20:31:32.928678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.928643    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvfp\" (UniqueName: \"kubernetes.io/projected/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-kube-api-access-6bvfp\") pod \"maas-api-revoke-sub-f14db2bd-jnzr2\" (UID: \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"
Nov 03 20:31:32.928846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.928759    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-jnzr2\" (UID: \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"
Nov 03 20:31:32.928911 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.928847    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-jnzr2\" (UID: \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"
Nov 03 20:31:32.929775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.929747    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-jnzr2\" (UID: \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"
Nov 03 20:31:32.931110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.931084    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-jnzr2\" (UID: \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"
Nov 03 20:31:32.940774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.940748    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvfp\" (UniqueName: \"kubernetes.io/projected/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-kube-api-access-6bvfp\") pod \"maas-api-revoke-sub-f14db2bd-jnzr2\" (UID: \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"
Nov 03 20:31:32.973142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:32.973113    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"
Nov 03 20:31:33.302906 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:31:33.302878    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a7bbce_13c8_4622_8af2_8eae3bad4f78.slice/crio-12fad0a06b96247039c3d9b936a77e2e72c6fefdfd28b4a403e58734cddc7ffb WatchSource:0}: Error finding container 12fad0a06b96247039c3d9b936a77e2e72c6fefdfd28b4a403e58734cddc7ffb: Status 404 returned error can't find the container with id 12fad0a06b96247039c3d9b936a77e2e72c6fefdfd28b4a403e58734cddc7ffb
Nov 03 20:31:33.304040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:33.304014    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"]
Nov 03 20:31:33.539001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:33.538974    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:33.540699 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:33.540673    5659 scope.go:122] "RemoveContainer" containerID="1bc66b4ad22257c3a190b44674829b850bf4d2880a03d37539b3bcd2e1c58c61"
Nov 03 20:31:33.541047 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:33.541025    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-n5gzh_odh-ai-gateway-infra(d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh" podUID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0"
Nov 03 20:31:33.542048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:33.542024    5659 generic.go:356] "Generic (PLEG): container finished" podID="025ae598-4df8-4574-a9b2-677c19a15fb7" containerID="0aa0f7d1dee3205ab95652f03353871041be1ec3f7990f70e392f15c9e576c2b" exitCode=22
Nov 03 20:31:33.542189 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:33.542165    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk" event={"ID":"025ae598-4df8-4574-a9b2-677c19a15fb7","Type":"ContainerDied","Data":"0aa0f7d1dee3205ab95652f03353871041be1ec3f7990f70e392f15c9e576c2b"}
Nov 03 20:31:33.542401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:33.542376    5659 scope.go:122] "RemoveContainer" containerID="0aa0f7d1dee3205ab95652f03353871041be1ec3f7990f70e392f15c9e576c2b"
Nov 03 20:31:33.544355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:33.544063    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2" event={"ID":"d1a7bbce-13c8-4622-8af2-8eae3bad4f78","Type":"ContainerStarted","Data":"12fad0a06b96247039c3d9b936a77e2e72c6fefdfd28b4a403e58734cddc7ffb"}
Nov 03 20:31:33.986456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:33.986426    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"
Nov 03 20:31:34.063055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.063015    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e214689-ceac-4756-9da8-586f6bbbac23-kube-api-access-vxb8s\" (UniqueName: \"kubernetes.io/projected/6e214689-ceac-4756-9da8-586f6bbbac23-kube-api-access-vxb8s\") pod \"6e214689-ceac-4756-9da8-586f6bbbac23\" (UID: \"6e214689-ceac-4756-9da8-586f6bbbac23\") "
Nov 03 20:31:34.063281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.063150    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-xks-ca\") pod \"6e214689-ceac-4756-9da8-586f6bbbac23\" (UID: \"6e214689-ceac-4756-9da8-586f6bbbac23\") "
Nov 03 20:31:34.063468 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.063449    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-openshift-ca\") pod \"6e214689-ceac-4756-9da8-586f6bbbac23\" (UID: \"6e214689-ceac-4756-9da8-586f6bbbac23\") "
Nov 03 20:31:34.064166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.064111    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-openshift-ca" pod "6e214689-ceac-4756-9da8-586f6bbbac23" (UID: "6e214689-ceac-4756-9da8-586f6bbbac23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:34.065729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.065708    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e214689-ceac-4756-9da8-586f6bbbac23-kube-api-access-vxb8s" pod "6e214689-ceac-4756-9da8-586f6bbbac23" (UID: "6e214689-ceac-4756-9da8-586f6bbbac23"). InnerVolumeSpecName "kube-api-access-vxb8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:34.066103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.066084    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-xks-ca" pod "6e214689-ceac-4756-9da8-586f6bbbac23" (UID: "6e214689-ceac-4756-9da8-586f6bbbac23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:34.170256 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.170214    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:34.170256 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.170255    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e214689-ceac-4756-9da8-586f6bbbac23-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:34.170590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.170272    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxb8s\" (UniqueName: \"kubernetes.io/projected/6e214689-ceac-4756-9da8-586f6bbbac23-kube-api-access-vxb8s\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:34.359590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.359548    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:34.555789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.555754    5659 generic.go:356] "Generic (PLEG): container finished" podID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78" containerID="1978a3588e4a659633cc4a2889de429757be9e13ea4504c5b9c9248f11973aae" exitCode=22
Nov 03 20:31:34.556228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.555815    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2" event={"ID":"d1a7bbce-13c8-4622-8af2-8eae3bad4f78","Type":"ContainerDied","Data":"1978a3588e4a659633cc4a2889de429757be9e13ea4504c5b9c9248f11973aae"}
Nov 03 20:31:34.556315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.556263    5659 scope.go:122] "RemoveContainer" containerID="1978a3588e4a659633cc4a2889de429757be9e13ea4504c5b9c9248f11973aae"
Nov 03 20:31:34.557735 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.557709    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"
Nov 03 20:31:34.557865 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.557745    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss" event={"ID":"6e214689-ceac-4756-9da8-586f6bbbac23","Type":"ContainerDied","Data":"84e64bec851f794a88748a59e59939e6d86671b22c16ef346d2fa24192efe9a2"}
Nov 03 20:31:34.557865 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.557799    5659 scope.go:122] "RemoveContainer" containerID="74c51b9a550d32444b6fecae53c37d3c49d101d8012a892b95e0f2253439658f"
Nov 03 20:31:34.560764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.560674    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk" event={"ID":"025ae598-4df8-4574-a9b2-677c19a15fb7","Type":"ContainerStarted","Data":"fcab96818331d1fdb9c070623f405bccf47d5be211ebc4306c82fcdd5786668b"}
Nov 03 20:31:34.596319 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.596281    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"]
Nov 03 20:31:34.601284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.601258    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-4vdss"]
Nov 03 20:31:34.962861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:34.962831    5659 scope.go:122] "RemoveContainer" containerID="7b4c6687c05f07a8b1d9aa25a8763a8719dd750d7709cf6465a949c641ace113"
Nov 03 20:31:35.048576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:35.048547    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:35.573398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:35.573361    5659 generic.go:356] "Generic (PLEG): container finished" podID="025ae598-4df8-4574-a9b2-677c19a15fb7" containerID="fcab96818331d1fdb9c070623f405bccf47d5be211ebc4306c82fcdd5786668b" exitCode=22
Nov 03 20:31:35.573898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:35.573441    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk" event={"ID":"025ae598-4df8-4574-a9b2-677c19a15fb7","Type":"ContainerDied","Data":"fcab96818331d1fdb9c070623f405bccf47d5be211ebc4306c82fcdd5786668b"}
Nov 03 20:31:35.573898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:35.573521    5659 scope.go:122] "RemoveContainer" containerID="0aa0f7d1dee3205ab95652f03353871041be1ec3f7990f70e392f15c9e576c2b"
Nov 03 20:31:35.573898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:35.573874    5659 scope.go:122] "RemoveContainer" containerID="fcab96818331d1fdb9c070623f405bccf47d5be211ebc4306c82fcdd5786668b"
Nov 03 20:31:35.574370 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:35.574342    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-jvbjk_odh-ai-gateway-infra(025ae598-4df8-4574-a9b2-677c19a15fb7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk" podUID="025ae598-4df8-4574-a9b2-677c19a15fb7"
Nov 03 20:31:35.575810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:35.575788    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8" event={"ID":"4afae507-7a10-43c4-92ef-864f1ecf3bde","Type":"ContainerStarted","Data":"6c9579f1847d9fc1da87b7f769355ac651c846370abe72cb70f5e703cd465ae4"}
Nov 03 20:31:35.578003 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:35.577978    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2" event={"ID":"d1a7bbce-13c8-4622-8af2-8eae3bad4f78","Type":"ContainerStarted","Data":"cd588620745a46b84ff21e254bb28b7136a2629f222fe022018cb943e7ea797d"}
Nov 03 20:31:35.638028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:35.637968    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2" podStartSLOduration=3.637942093 podStartE2EDuration="3.637942093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:35.634097829 +0000 UTC m=+3247.789000249" watchObservedRunningTime="2026-11-03 20:31:35.637942093 +0000 UTC m=+3247.792844513"
Nov 03 20:31:35.849115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:35.849060    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:35.976157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:35.976120    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e214689-ceac-4756-9da8-586f6bbbac23" path="/var/lib/kubelet/pods/6e214689-ceac-4756-9da8-586f6bbbac23/volumes"
Nov 03 20:31:36.557520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:36.557294    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:36.614504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:36.605466    5659 scope.go:122] "RemoveContainer" containerID="fcab96818331d1fdb9c070623f405bccf47d5be211ebc4306c82fcdd5786668b"
Nov 03 20:31:36.614504 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:36.606131    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-jvbjk_odh-ai-gateway-infra(025ae598-4df8-4574-a9b2-677c19a15fb7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk" podUID="025ae598-4df8-4574-a9b2-677c19a15fb7"
Nov 03 20:31:36.619503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:36.615999    5659 generic.go:356] "Generic (PLEG): container finished" podID="4afae507-7a10-43c4-92ef-864f1ecf3bde" containerID="6c9579f1847d9fc1da87b7f769355ac651c846370abe72cb70f5e703cd465ae4" exitCode=22
Nov 03 20:31:36.619503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:36.616205    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8" event={"ID":"4afae507-7a10-43c4-92ef-864f1ecf3bde","Type":"ContainerDied","Data":"6c9579f1847d9fc1da87b7f769355ac651c846370abe72cb70f5e703cd465ae4"}
Nov 03 20:31:36.619503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:36.616252    5659 scope.go:122] "RemoveContainer" containerID="7b4c6687c05f07a8b1d9aa25a8763a8719dd750d7709cf6465a949c641ace113"
Nov 03 20:31:36.619503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:36.616846    5659 scope.go:122] "RemoveContainer" containerID="6c9579f1847d9fc1da87b7f769355ac651c846370abe72cb70f5e703cd465ae4"
Nov 03 20:31:36.619503 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:36.617415    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-zj9s8_odh-ai-gateway-infra(4afae507-7a10-43c4-92ef-864f1ecf3bde)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8" podUID="4afae507-7a10-43c4-92ef-864f1ecf3bde"
Nov 03 20:31:36.627517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:36.621117    5659 generic.go:356] "Generic (PLEG): container finished" podID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78" containerID="cd588620745a46b84ff21e254bb28b7136a2629f222fe022018cb943e7ea797d" exitCode=22
Nov 03 20:31:36.627517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:36.621179    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2" event={"ID":"d1a7bbce-13c8-4622-8af2-8eae3bad4f78","Type":"ContainerDied","Data":"cd588620745a46b84ff21e254bb28b7136a2629f222fe022018cb943e7ea797d"}
Nov 03 20:31:36.627517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:36.621698    5659 scope.go:122] "RemoveContainer" containerID="cd588620745a46b84ff21e254bb28b7136a2629f222fe022018cb943e7ea797d"
Nov 03 20:31:36.627517 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:36.622203    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-jnzr2_odh-ai-gateway-infra(d1a7bbce-13c8-4622-8af2-8eae3bad4f78)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2" podUID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78"
Nov 03 20:31:36.640619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:36.640582    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"]
Nov 03 20:31:36.650206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:36.650179    5659 scope.go:122] "RemoveContainer" containerID="1978a3588e4a659633cc4a2889de429757be9e13ea4504c5b9c9248f11973aae"
Nov 03 20:31:37.429494 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:37.429447    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:37.633999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:37.633970    5659 scope.go:122] "RemoveContainer" containerID="cd588620745a46b84ff21e254bb28b7136a2629f222fe022018cb943e7ea797d"
Nov 03 20:31:37.634542 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:37.634412    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-jnzr2_odh-ai-gateway-infra(d1a7bbce-13c8-4622-8af2-8eae3bad4f78)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2" podUID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78"
Nov 03 20:31:37.847113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:37.847083    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"
Nov 03 20:31:37.961694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:37.961606    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-openshift-ca\") pod \"4afae507-7a10-43c4-92ef-864f1ecf3bde\" (UID: \"4afae507-7a10-43c4-92ef-864f1ecf3bde\") "
Nov 03 20:31:37.961866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:37.961697    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4afae507-7a10-43c4-92ef-864f1ecf3bde-kube-api-access-mhthr\" (UniqueName: \"kubernetes.io/projected/4afae507-7a10-43c4-92ef-864f1ecf3bde-kube-api-access-mhthr\") pod \"4afae507-7a10-43c4-92ef-864f1ecf3bde\" (UID: \"4afae507-7a10-43c4-92ef-864f1ecf3bde\") "
Nov 03 20:31:37.961946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:37.961862    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-xks-ca\") pod \"4afae507-7a10-43c4-92ef-864f1ecf3bde\" (UID: \"4afae507-7a10-43c4-92ef-864f1ecf3bde\") "
Nov 03 20:31:37.965046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:37.964928    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-openshift-ca" pod "4afae507-7a10-43c4-92ef-864f1ecf3bde" (UID: "4afae507-7a10-43c4-92ef-864f1ecf3bde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:37.966293 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:37.966270    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-xks-ca" pod "4afae507-7a10-43c4-92ef-864f1ecf3bde" (UID: "4afae507-7a10-43c4-92ef-864f1ecf3bde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:37.968902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:37.968867    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afae507-7a10-43c4-92ef-864f1ecf3bde-kube-api-access-mhthr" pod "4afae507-7a10-43c4-92ef-864f1ecf3bde" (UID: "4afae507-7a10-43c4-92ef-864f1ecf3bde"). InnerVolumeSpecName "kube-api-access-mhthr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:38.021563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:38.021525    5659 scope.go:122] "RemoveContainer" containerID="fd613968f1fcebaa41b0cf19e87f877588d81d5f963bbc5eb609dad4b0ca5dd0"
Nov 03 20:31:38.065801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:38.065766    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:38.065938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:38.065815    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhthr\" (UniqueName: \"kubernetes.io/projected/4afae507-7a10-43c4-92ef-864f1ecf3bde-kube-api-access-mhthr\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:38.065938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:38.065834    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afae507-7a10-43c4-92ef-864f1ecf3bde-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:38.648939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:38.648837    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8" event={"ID":"4afae507-7a10-43c4-92ef-864f1ecf3bde","Type":"ContainerDied","Data":"8bf662f6018954176badcbaf60bcce16e5f22afb9a75d5717adde63973db975b"}
Nov 03 20:31:38.648939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:38.648915    5659 scope.go:122] "RemoveContainer" containerID="6c9579f1847d9fc1da87b7f769355ac651c846370abe72cb70f5e703cd465ae4"
Nov 03 20:31:38.649446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:38.649048    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"
Nov 03 20:31:38.654024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:38.653993    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx" event={"ID":"6724b388-74ed-45a2-b283-f64245c0a06a","Type":"ContainerStarted","Data":"3b02160e78eacff0fdaa15633510be7601bb8e7586ffce8cb605b9afe6713f1b"}
Nov 03 20:31:38.673739 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:38.673707    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"]
Nov 03 20:31:38.678822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:38.678797    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-zj9s8"]
Nov 03 20:31:39.181506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:39.179213    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:39.320626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:39.320589    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:39.668034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:39.667993    5659 generic.go:356] "Generic (PLEG): container finished" podID="6724b388-74ed-45a2-b283-f64245c0a06a" containerID="3b02160e78eacff0fdaa15633510be7601bb8e7586ffce8cb605b9afe6713f1b" exitCode=22
Nov 03 20:31:39.668638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:39.668035    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx" event={"ID":"6724b388-74ed-45a2-b283-f64245c0a06a","Type":"ContainerDied","Data":"3b02160e78eacff0fdaa15633510be7601bb8e7586ffce8cb605b9afe6713f1b"}
Nov 03 20:31:39.668638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:39.668101    5659 scope.go:122] "RemoveContainer" containerID="fd613968f1fcebaa41b0cf19e87f877588d81d5f963bbc5eb609dad4b0ca5dd0"
Nov 03 20:31:39.668638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:39.668580    5659 scope.go:122] "RemoveContainer" containerID="3b02160e78eacff0fdaa15633510be7601bb8e7586ffce8cb605b9afe6713f1b"
Nov 03 20:31:39.669107 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:39.669058    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-4whvx_odh-ai-gateway-infra(6724b388-74ed-45a2-b283-f64245c0a06a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx" podUID="6724b388-74ed-45a2-b283-f64245c0a06a"
Nov 03 20:31:39.709181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:39.709147    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"]
Nov 03 20:31:39.975139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:39.975098    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4afae507-7a10-43c4-92ef-864f1ecf3bde" path="/var/lib/kubelet/pods/4afae507-7a10-43c4-92ef-864f1ecf3bde/volumes"
Nov 03 20:31:39.989359 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:39.989297    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:40.840591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:40.840385    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:40.964307 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:40.964272    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"
Nov 03 20:31:41.041789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.041753    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-openshift-ca\") pod \"6724b388-74ed-45a2-b283-f64245c0a06a\" (UID: \"6724b388-74ed-45a2-b283-f64245c0a06a\") "
Nov 03 20:31:41.042003 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.041923    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-xks-ca\") pod \"6724b388-74ed-45a2-b283-f64245c0a06a\" (UID: \"6724b388-74ed-45a2-b283-f64245c0a06a\") "
Nov 03 20:31:41.042086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.042008    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6724b388-74ed-45a2-b283-f64245c0a06a-kube-api-access-dvwfq\" (UniqueName: \"kubernetes.io/projected/6724b388-74ed-45a2-b283-f64245c0a06a-kube-api-access-dvwfq\") pod \"6724b388-74ed-45a2-b283-f64245c0a06a\" (UID: \"6724b388-74ed-45a2-b283-f64245c0a06a\") "
Nov 03 20:31:41.044704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.044670    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-openshift-ca" pod "6724b388-74ed-45a2-b283-f64245c0a06a" (UID: "6724b388-74ed-45a2-b283-f64245c0a06a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:41.045858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.045827    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6724b388-74ed-45a2-b283-f64245c0a06a-kube-api-access-dvwfq" pod "6724b388-74ed-45a2-b283-f64245c0a06a" (UID: "6724b388-74ed-45a2-b283-f64245c0a06a"). InnerVolumeSpecName "kube-api-access-dvwfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:41.049890 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.049866    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-xks-ca" pod "6724b388-74ed-45a2-b283-f64245c0a06a" (UID: "6724b388-74ed-45a2-b283-f64245c0a06a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:41.145388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.145347    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:41.145388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.145386    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6724b388-74ed-45a2-b283-f64245c0a06a-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:41.145637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.145404    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvwfq\" (UniqueName: \"kubernetes.io/projected/6724b388-74ed-45a2-b283-f64245c0a06a-kube-api-access-dvwfq\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:41.446533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.444162    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:41.691659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.691630    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"
Nov 03 20:31:41.691840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.691661    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx" event={"ID":"6724b388-74ed-45a2-b283-f64245c0a06a","Type":"ContainerDied","Data":"7df7a5acbcfb1bfb5c12428ff76741c62962d87352914203d00fb2e2dffda1d6"}
Nov 03 20:31:41.691840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.691721    5659 scope.go:122] "RemoveContainer" containerID="3b02160e78eacff0fdaa15633510be7601bb8e7586ffce8cb605b9afe6713f1b"
Nov 03 20:31:41.718673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.718643    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"]
Nov 03 20:31:41.724328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.724301    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-4whvx"]
Nov 03 20:31:41.982029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:41.981990    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6724b388-74ed-45a2-b283-f64245c0a06a" path="/var/lib/kubelet/pods/6724b388-74ed-45a2-b283-f64245c0a06a/volumes"
Nov 03 20:31:42.155284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.155247    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:42.265255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.265158    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"]
Nov 03 20:31:42.270363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.270316    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6724b388-74ed-45a2-b283-f64245c0a06a" containerName="revoke-keys"
Nov 03 20:31:42.272573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.272543    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6724b388-74ed-45a2-b283-f64245c0a06a" containerName="revoke-keys"
Nov 03 20:31:42.272573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.272576    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6e214689-ceac-4756-9da8-586f6bbbac23" containerName="revoke-keys"
Nov 03 20:31:42.272573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.272585    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e214689-ceac-4756-9da8-586f6bbbac23" containerName="revoke-keys"
Nov 03 20:31:42.272842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.272596    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6e214689-ceac-4756-9da8-586f6bbbac23" containerName="revoke-keys"
Nov 03 20:31:42.272842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.272603    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e214689-ceac-4756-9da8-586f6bbbac23" containerName="revoke-keys"
Nov 03 20:31:42.272842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.272619    5659 cpu_manager.go:414] "containerMap: removing container" podUID="4afae507-7a10-43c4-92ef-864f1ecf3bde" containerName="revoke-keys"
Nov 03 20:31:42.272842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.272628    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4afae507-7a10-43c4-92ef-864f1ecf3bde" containerName="revoke-keys"
Nov 03 20:31:42.272842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.272649    5659 cpu_manager.go:414] "containerMap: removing container" podUID="4afae507-7a10-43c4-92ef-864f1ecf3bde" containerName="revoke-keys"
Nov 03 20:31:42.272842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.272656    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4afae507-7a10-43c4-92ef-864f1ecf3bde" containerName="revoke-keys"
Nov 03 20:31:42.272842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.272674    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6724b388-74ed-45a2-b283-f64245c0a06a" containerName="revoke-keys"
Nov 03 20:31:42.272842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.272680    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6724b388-74ed-45a2-b283-f64245c0a06a" containerName="revoke-keys"
Nov 03 20:31:42.273254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.272993    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6724b388-74ed-45a2-b283-f64245c0a06a" containerName="revoke-keys"
Nov 03 20:31:42.273254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.273010    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e214689-ceac-4756-9da8-586f6bbbac23" containerName="revoke-keys"
Nov 03 20:31:42.273254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.273030    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="4afae507-7a10-43c4-92ef-864f1ecf3bde" containerName="revoke-keys"
Nov 03 20:31:42.273254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.273043    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e214689-ceac-4756-9da8-586f6bbbac23" containerName="revoke-keys"
Nov 03 20:31:42.273254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.273052    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e214689-ceac-4756-9da8-586f6bbbac23" containerName="revoke-keys"
Nov 03 20:31:42.273254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.273063    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="4afae507-7a10-43c4-92ef-864f1ecf3bde" containerName="revoke-keys"
Nov 03 20:31:42.273254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.273078    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6724b388-74ed-45a2-b283-f64245c0a06a" containerName="revoke-keys"
Nov 03 20:31:42.317167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.315918    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"]
Nov 03 20:31:42.317167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.316085    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"
Nov 03 20:31:42.373603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.373558    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-ccfxb\" (UID: \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"
Nov 03 20:31:42.373809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.373611    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhxb5\" (UniqueName: \"kubernetes.io/projected/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-kube-api-access-nhxb5\") pod \"maas-api-revoke-sub-abad4031-ccfxb\" (UID: \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"
Nov 03 20:31:42.374015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.373996    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-ccfxb\" (UID: \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"
Nov 03 20:31:42.475414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.475374    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-ccfxb\" (UID: \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"
Nov 03 20:31:42.475651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.475436    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-ccfxb\" (UID: \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"
Nov 03 20:31:42.475651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.475463    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhxb5\" (UniqueName: \"kubernetes.io/projected/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-kube-api-access-nhxb5\") pod \"maas-api-revoke-sub-abad4031-ccfxb\" (UID: \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"
Nov 03 20:31:42.476280 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.476248    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-ccfxb\" (UID: \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"
Nov 03 20:31:42.478212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.478190    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-ccfxb\" (UID: \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"
Nov 03 20:31:42.487893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.487868    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhxb5\" (UniqueName: \"kubernetes.io/projected/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-kube-api-access-nhxb5\") pod \"maas-api-revoke-sub-abad4031-ccfxb\" (UID: \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"
Nov 03 20:31:42.640378 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.640290    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"
Nov 03 20:31:42.765778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.765739    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"]
Nov 03 20:31:42.767475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.767332    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6e214689-ceac-4756-9da8-586f6bbbac23" containerName="revoke-keys"
Nov 03 20:31:42.767475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.767356    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e214689-ceac-4756-9da8-586f6bbbac23" containerName="revoke-keys"
Nov 03 20:31:42.767475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.767380    5659 cpu_manager.go:414] "containerMap: removing container" podUID="4afae507-7a10-43c4-92ef-864f1ecf3bde" containerName="revoke-keys"
Nov 03 20:31:42.767475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.767388    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4afae507-7a10-43c4-92ef-864f1ecf3bde" containerName="revoke-keys"
Nov 03 20:31:42.767475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.767407    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6724b388-74ed-45a2-b283-f64245c0a06a" containerName="revoke-keys"
Nov 03 20:31:42.767475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.767415    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6724b388-74ed-45a2-b283-f64245c0a06a" containerName="revoke-keys"
Nov 03 20:31:42.768682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.767694    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="4afae507-7a10-43c4-92ef-864f1ecf3bde" containerName="revoke-keys"
Nov 03 20:31:42.768682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.767721    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6724b388-74ed-45a2-b283-f64245c0a06a" containerName="revoke-keys"
Nov 03 20:31:42.775587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.775169    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"
Nov 03 20:31:42.781573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.781464    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"]
Nov 03 20:31:42.783499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.783461    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-tg7tv\" (UID: \"b1050e12-dfb8-41df-876f-c754f3567fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"
Nov 03 20:31:42.783663 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.783643    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-tg7tv\" (UID: \"b1050e12-dfb8-41df-876f-c754f3567fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"
Nov 03 20:31:42.785387 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.783756    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr4fs\" (UniqueName: \"kubernetes.io/projected/b1050e12-dfb8-41df-876f-c754f3567fc5-kube-api-access-vr4fs\") pod \"maas-api-revoke-sub-5bac6f1e-tg7tv\" (UID: \"b1050e12-dfb8-41df-876f-c754f3567fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"
Nov 03 20:31:42.884677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.884640    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-tg7tv\" (UID: \"b1050e12-dfb8-41df-876f-c754f3567fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"
Nov 03 20:31:42.884897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.884764    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-tg7tv\" (UID: \"b1050e12-dfb8-41df-876f-c754f3567fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"
Nov 03 20:31:42.884897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.884867    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr4fs\" (UniqueName: \"kubernetes.io/projected/b1050e12-dfb8-41df-876f-c754f3567fc5-kube-api-access-vr4fs\") pod \"maas-api-revoke-sub-5bac6f1e-tg7tv\" (UID: \"b1050e12-dfb8-41df-876f-c754f3567fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"
Nov 03 20:31:42.885461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.885437    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-tg7tv\" (UID: \"b1050e12-dfb8-41df-876f-c754f3567fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"
Nov 03 20:31:42.887177 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.887156    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-tg7tv\" (UID: \"b1050e12-dfb8-41df-876f-c754f3567fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"
Nov 03 20:31:42.896892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.896836    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr4fs\" (UniqueName: \"kubernetes.io/projected/b1050e12-dfb8-41df-876f-c754f3567fc5-kube-api-access-vr4fs\") pod \"maas-api-revoke-sub-5bac6f1e-tg7tv\" (UID: \"b1050e12-dfb8-41df-876f-c754f3567fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"
Nov 03 20:31:42.951225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:42.950386    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"]
Nov 03 20:31:42.952063 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:31:42.952034    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedbbc3ca_4ab5_46af_868d_c1f3a50bf3ef.slice/crio-0a5aa747bdc5702e9d596c852941e2a425525983b6705045dc31d695e37742f5 WatchSource:0}: Error finding container 0a5aa747bdc5702e9d596c852941e2a425525983b6705045dc31d695e37742f5: Status 404 returned error can't find the container with id 0a5aa747bdc5702e9d596c852941e2a425525983b6705045dc31d695e37742f5
Nov 03 20:31:43.099546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:43.099508    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"
Nov 03 20:31:43.190471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:43.190373    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:43.486660 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:31:43.486626    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1050e12_dfb8_41df_876f_c754f3567fc5.slice/crio-0576500aabb73b819720e1a359553ad01cbeb7de154bbf34f190f48d61c48674 WatchSource:0}: Error finding container 0576500aabb73b819720e1a359553ad01cbeb7de154bbf34f190f48d61c48674: Status 404 returned error can't find the container with id 0576500aabb73b819720e1a359553ad01cbeb7de154bbf34f190f48d61c48674
Nov 03 20:31:43.487654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:43.487627    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"]
Nov 03 20:31:43.716369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:43.716329    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb" event={"ID":"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef","Type":"ContainerStarted","Data":"3a85ef276afa24f213eaf515d5dd5fb5bb1a8e39cff0319ef2b5771061393cfc"}
Nov 03 20:31:43.716584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:43.716400    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb" event={"ID":"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef","Type":"ContainerStarted","Data":"0a5aa747bdc5702e9d596c852941e2a425525983b6705045dc31d695e37742f5"}
Nov 03 20:31:43.717965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:43.717939    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv" event={"ID":"b1050e12-dfb8-41df-876f-c754f3567fc5","Type":"ContainerStarted","Data":"0576500aabb73b819720e1a359553ad01cbeb7de154bbf34f190f48d61c48674"}
Nov 03 20:31:43.739466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:43.739414    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb" podStartSLOduration=1.739398357 podStartE2EDuration="1.739398357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:43.734047881 +0000 UTC m=+3255.888950302" watchObservedRunningTime="2026-11-03 20:31:43.739398357 +0000 UTC m=+3255.894300776"
Nov 03 20:31:43.843383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:43.843350    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:44.549821 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:44.547288    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:44.728527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:44.728461    5659 generic.go:356] "Generic (PLEG): container finished" podID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" containerID="3a85ef276afa24f213eaf515d5dd5fb5bb1a8e39cff0319ef2b5771061393cfc" exitCode=22
Nov 03 20:31:44.728712 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:44.728538    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb" event={"ID":"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef","Type":"ContainerDied","Data":"3a85ef276afa24f213eaf515d5dd5fb5bb1a8e39cff0319ef2b5771061393cfc"}
Nov 03 20:31:44.728893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:44.728873    5659 scope.go:122] "RemoveContainer" containerID="3a85ef276afa24f213eaf515d5dd5fb5bb1a8e39cff0319ef2b5771061393cfc"
Nov 03 20:31:44.730640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:44.730618    5659 generic.go:356] "Generic (PLEG): container finished" podID="b1050e12-dfb8-41df-876f-c754f3567fc5" containerID="5f1335e0e458c8fcc20650435bdcf6e85771f6968591b2617fafa80e657dbde0" exitCode=22
Nov 03 20:31:44.730750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:44.730727    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv" event={"ID":"b1050e12-dfb8-41df-876f-c754f3567fc5","Type":"ContainerDied","Data":"5f1335e0e458c8fcc20650435bdcf6e85771f6968591b2617fafa80e657dbde0"}
Nov 03 20:31:44.730966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:44.730946    5659 scope.go:122] "RemoveContainer" containerID="5f1335e0e458c8fcc20650435bdcf6e85771f6968591b2617fafa80e657dbde0"
Nov 03 20:31:45.219973 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:45.219939    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:31:45.453707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:45.453674    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:45.769198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:45.769143    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb" event={"ID":"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef","Type":"ContainerStarted","Data":"a0afbe7257d8cc4cc6bf34438d38ad1235aa1fc9ac298f440219e318e1104251"}
Nov 03 20:31:45.771409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:45.771380    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv" event={"ID":"b1050e12-dfb8-41df-876f-c754f3567fc5","Type":"ContainerStarted","Data":"35a9dd8133650ea152120cdf310b246b72f7f037b87810be66950247db0b0772"}
Nov 03 20:31:45.807056 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:45.807002    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv" podStartSLOduration=3.8069877549999998 podStartE2EDuration="3.806987755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:45.805800398 +0000 UTC m=+3257.960702817" watchObservedRunningTime="2026-11-03 20:31:45.806987755 +0000 UTC m=+3257.961890174"
Nov 03 20:31:45.961657 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:45.961622    5659 scope.go:122] "RemoveContainer" containerID="1bc66b4ad22257c3a190b44674829b850bf4d2880a03d37539b3bcd2e1c58c61"
Nov 03 20:31:46.188655 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.188616    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:46.441669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.441638    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"]
Nov 03 20:31:46.447101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.447069    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"
Nov 03 20:31:46.453341 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.453312    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"]
Nov 03 20:31:46.477268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.477229    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-s5pjw\" (UID: \"6fde0783-085d-44fe-a55e-18700eaa6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"
Nov 03 20:31:46.477458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.477423    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbknp\" (UniqueName: \"kubernetes.io/projected/6fde0783-085d-44fe-a55e-18700eaa6e88-kube-api-access-zbknp\") pod \"maas-api-revoke-sub-266d0609-s5pjw\" (UID: \"6fde0783-085d-44fe-a55e-18700eaa6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"
Nov 03 20:31:46.477672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.477496    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-s5pjw\" (UID: \"6fde0783-085d-44fe-a55e-18700eaa6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"
Nov 03 20:31:46.579304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.579207    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-s5pjw\" (UID: \"6fde0783-085d-44fe-a55e-18700eaa6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"
Nov 03 20:31:46.579504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.579356    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbknp\" (UniqueName: \"kubernetes.io/projected/6fde0783-085d-44fe-a55e-18700eaa6e88-kube-api-access-zbknp\") pod \"maas-api-revoke-sub-266d0609-s5pjw\" (UID: \"6fde0783-085d-44fe-a55e-18700eaa6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"
Nov 03 20:31:46.579504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.579396    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-s5pjw\" (UID: \"6fde0783-085d-44fe-a55e-18700eaa6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"
Nov 03 20:31:46.580048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.580027    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-s5pjw\" (UID: \"6fde0783-085d-44fe-a55e-18700eaa6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"
Nov 03 20:31:46.581950 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.581924    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-s5pjw\" (UID: \"6fde0783-085d-44fe-a55e-18700eaa6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"
Nov 03 20:31:46.591769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.591745    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbknp\" (UniqueName: \"kubernetes.io/projected/6fde0783-085d-44fe-a55e-18700eaa6e88-kube-api-access-zbknp\") pod \"maas-api-revoke-sub-266d0609-s5pjw\" (UID: \"6fde0783-085d-44fe-a55e-18700eaa6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"
Nov 03 20:31:46.778431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.778395    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"
Nov 03 20:31:46.783180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.783148    5659 generic.go:356] "Generic (PLEG): container finished" podID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" containerID="a0afbe7257d8cc4cc6bf34438d38ad1235aa1fc9ac298f440219e318e1104251" exitCode=22
Nov 03 20:31:46.783333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.783238    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb" event={"ID":"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef","Type":"ContainerDied","Data":"a0afbe7257d8cc4cc6bf34438d38ad1235aa1fc9ac298f440219e318e1104251"}
Nov 03 20:31:46.783333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.783313    5659 scope.go:122] "RemoveContainer" containerID="3a85ef276afa24f213eaf515d5dd5fb5bb1a8e39cff0319ef2b5771061393cfc"
Nov 03 20:31:46.783542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.783521    5659 scope.go:122] "RemoveContainer" containerID="a0afbe7257d8cc4cc6bf34438d38ad1235aa1fc9ac298f440219e318e1104251"
Nov 03 20:31:46.783809 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:46.783787    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-ccfxb_odh-ai-gateway-infra(edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb" podUID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef"
Nov 03 20:31:46.785554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.785525    5659 generic.go:356] "Generic (PLEG): container finished" podID="b1050e12-dfb8-41df-876f-c754f3567fc5" containerID="35a9dd8133650ea152120cdf310b246b72f7f037b87810be66950247db0b0772" exitCode=22
Nov 03 20:31:46.785660 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.785612    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv" event={"ID":"b1050e12-dfb8-41df-876f-c754f3567fc5","Type":"ContainerDied","Data":"35a9dd8133650ea152120cdf310b246b72f7f037b87810be66950247db0b0772"}
Nov 03 20:31:46.785812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.785798    5659 scope.go:122] "RemoveContainer" containerID="35a9dd8133650ea152120cdf310b246b72f7f037b87810be66950247db0b0772"
Nov 03 20:31:46.786049 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:46.786027    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-tg7tv_odh-ai-gateway-infra(b1050e12-dfb8-41df-876f-c754f3567fc5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv" podUID="b1050e12-dfb8-41df-876f-c754f3567fc5"
Nov 03 20:31:46.787722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.787699    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh" event={"ID":"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0","Type":"ContainerStarted","Data":"3614db98f4b549487754e0a774a29cf538ecbfd8beb6c9595317c65f6f296e62"}
Nov 03 20:31:46.802878 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.802853    5659 scope.go:122] "RemoveContainer" containerID="5f1335e0e458c8fcc20650435bdcf6e85771f6968591b2617fafa80e657dbde0"
Nov 03 20:31:46.901678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.901646    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:46.975659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:46.975630    5659 scope.go:122] "RemoveContainer" containerID="cd588620745a46b84ff21e254bb28b7136a2629f222fe022018cb943e7ea797d"
Nov 03 20:31:47.090185 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:47.089954    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"]
Nov 03 20:31:47.093688 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:31:47.093651    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fde0783_085d_44fe_a55e_18700eaa6e88.slice/crio-57715ad7069648978d86045b31f9ffc0cf478deeb9b42a5ee40f06e0000cf02a WatchSource:0}: Error finding container 57715ad7069648978d86045b31f9ffc0cf478deeb9b42a5ee40f06e0000cf02a: Status 404 returned error can't find the container with id 57715ad7069648978d86045b31f9ffc0cf478deeb9b42a5ee40f06e0000cf02a
Nov 03 20:31:47.668354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:47.663181    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:47.805671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:47.805590    5659 scope.go:122] "RemoveContainer" containerID="35a9dd8133650ea152120cdf310b246b72f7f037b87810be66950247db0b0772"
Nov 03 20:31:47.806157 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:47.806064    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-tg7tv_odh-ai-gateway-infra(b1050e12-dfb8-41df-876f-c754f3567fc5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv" podUID="b1050e12-dfb8-41df-876f-c754f3567fc5"
Nov 03 20:31:47.807627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:47.807594    5659 generic.go:356] "Generic (PLEG): container finished" podID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" containerID="3614db98f4b549487754e0a774a29cf538ecbfd8beb6c9595317c65f6f296e62" exitCode=7
Nov 03 20:31:47.807771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:47.807607    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh" event={"ID":"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0","Type":"ContainerDied","Data":"3614db98f4b549487754e0a774a29cf538ecbfd8beb6c9595317c65f6f296e62"}
Nov 03 20:31:47.807771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:47.807756    5659 scope.go:122] "RemoveContainer" containerID="1bc66b4ad22257c3a190b44674829b850bf4d2880a03d37539b3bcd2e1c58c61"
Nov 03 20:31:47.808152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:47.808136    5659 scope.go:122] "RemoveContainer" containerID="3614db98f4b549487754e0a774a29cf538ecbfd8beb6c9595317c65f6f296e62"
Nov 03 20:31:47.808634 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:47.808587    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-n5gzh_odh-ai-gateway-infra(d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh" podUID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0"
Nov 03 20:31:47.809944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:47.809920    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw" event={"ID":"6fde0783-085d-44fe-a55e-18700eaa6e88","Type":"ContainerStarted","Data":"3382b54b90aa0b23f8d983a4707961ee76aba0b2efe07ce30b559039626b7c09"}
Nov 03 20:31:47.810087 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:47.809972    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw" event={"ID":"6fde0783-085d-44fe-a55e-18700eaa6e88","Type":"ContainerStarted","Data":"57715ad7069648978d86045b31f9ffc0cf478deeb9b42a5ee40f06e0000cf02a"}
Nov 03 20:31:47.812226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:47.812205    5659 scope.go:122] "RemoveContainer" containerID="a0afbe7257d8cc4cc6bf34438d38ad1235aa1fc9ac298f440219e318e1104251"
Nov 03 20:31:47.812608 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:47.812578    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-ccfxb_odh-ai-gateway-infra(edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb" podUID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef"
Nov 03 20:31:47.814221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:47.814200    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2" event={"ID":"d1a7bbce-13c8-4622-8af2-8eae3bad4f78","Type":"ContainerStarted","Data":"119d43a72646c5af498121c1d0cdc2e0874050522d72f808284cf5aae7e45cbc"}
Nov 03 20:31:47.862709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:47.862673    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"]
Nov 03 20:31:47.914400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:47.914329    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw" podStartSLOduration=1.914309475 podStartE2EDuration="1.914309475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:47.907067786 +0000 UTC m=+3260.061970207" watchObservedRunningTime="2026-11-03 20:31:47.914309475 +0000 UTC m=+3260.069211896"
Nov 03 20:31:48.437168 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:48.437130    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:48.826610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:48.826577    5659 generic.go:356] "Generic (PLEG): container finished" podID="6fde0783-085d-44fe-a55e-18700eaa6e88" containerID="3382b54b90aa0b23f8d983a4707961ee76aba0b2efe07ce30b559039626b7c09" exitCode=22
Nov 03 20:31:48.826996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:48.826663    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw" event={"ID":"6fde0783-085d-44fe-a55e-18700eaa6e88","Type":"ContainerDied","Data":"3382b54b90aa0b23f8d983a4707961ee76aba0b2efe07ce30b559039626b7c09"}
Nov 03 20:31:48.826996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:48.826908    5659 scope.go:122] "RemoveContainer" containerID="3382b54b90aa0b23f8d983a4707961ee76aba0b2efe07ce30b559039626b7c09"
Nov 03 20:31:48.829438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:48.829414    5659 generic.go:356] "Generic (PLEG): container finished" podID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78" containerID="119d43a72646c5af498121c1d0cdc2e0874050522d72f808284cf5aae7e45cbc" exitCode=22
Nov 03 20:31:48.829592 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:48.829519    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2" event={"ID":"d1a7bbce-13c8-4622-8af2-8eae3bad4f78","Type":"ContainerDied","Data":"119d43a72646c5af498121c1d0cdc2e0874050522d72f808284cf5aae7e45cbc"}
Nov 03 20:31:48.829674 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:48.829591    5659 scope.go:122] "RemoveContainer" containerID="cd588620745a46b84ff21e254bb28b7136a2629f222fe022018cb943e7ea797d"
Nov 03 20:31:48.829970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:48.829952    5659 scope.go:122] "RemoveContainer" containerID="119d43a72646c5af498121c1d0cdc2e0874050522d72f808284cf5aae7e45cbc"
Nov 03 20:31:48.831023 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:48.830927    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-jnzr2_odh-ai-gateway-infra(d1a7bbce-13c8-4622-8af2-8eae3bad4f78)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2" podUID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78"
Nov 03 20:31:48.892453 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:48.892415    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"]
Nov 03 20:31:48.970507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:48.963305    5659 scope.go:122] "RemoveContainer" containerID="fcab96818331d1fdb9c070623f405bccf47d5be211ebc4306c82fcdd5786668b"
Nov 03 20:31:49.081663 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.081600    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"
Nov 03 20:31:49.121153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.121122    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-kube-api-access-tmq52\" (UniqueName: \"kubernetes.io/projected/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-kube-api-access-tmq52\") pod \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\" (UID: \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\") "
Nov 03 20:31:49.121331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.121249    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-openshift-ca\") pod \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\" (UID: \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\") "
Nov 03 20:31:49.121331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.121297    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-xks-ca\") pod \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\" (UID: \"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0\") "
Nov 03 20:31:49.121669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.121640    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-openshift-ca" pod "d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" (UID: "d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:49.122330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.122305    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:49.123598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.123571    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-xks-ca" pod "d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" (UID: "d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:49.123904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.123883    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-kube-api-access-tmq52" pod "d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" (UID: "d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0"). InnerVolumeSpecName "kube-api-access-tmq52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:49.207157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.207062    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:49.228887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.228847    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmq52\" (UniqueName: \"kubernetes.io/projected/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-kube-api-access-tmq52\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:49.234607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.232468    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:49.842286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.842256    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"
Nov 03 20:31:49.842852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.842294    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh" event={"ID":"d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0","Type":"ContainerDied","Data":"b1d49eddf548687b560c829fabdca390a44b8d4cb5799419888283f2cf95e34b"}
Nov 03 20:31:49.842852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.842370    5659 scope.go:122] "RemoveContainer" containerID="3614db98f4b549487754e0a774a29cf538ecbfd8beb6c9595317c65f6f296e62"
Nov 03 20:31:49.851592 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.851563    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:49.852212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.852182    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk" event={"ID":"025ae598-4df8-4574-a9b2-677c19a15fb7","Type":"ContainerStarted","Data":"156914f3a1099c6d26fe0a7599ea849db0162ed73540e3d596e7f7761f59eb21"}
Nov 03 20:31:49.857522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.855097    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw" event={"ID":"6fde0783-085d-44fe-a55e-18700eaa6e88","Type":"ContainerStarted","Data":"1a1707c2faecd88955e272ad167547d91353950a323a5c49e29d2072086ceac6"}
Nov 03 20:31:49.986188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.986154    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"]
Nov 03 20:31:49.986364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:49.986202    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-n5gzh"]
Nov 03 20:31:50.235998 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.235964    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"
Nov 03 20:31:50.276133 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.276093    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-openshift-ca\") pod \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\" (UID: \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\") "
Nov 03 20:31:50.276351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.276322    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-xks-ca\") pod \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\" (UID: \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\") "
Nov 03 20:31:50.276431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.276414    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-kube-api-access-6bvfp\" (UniqueName: \"kubernetes.io/projected/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-kube-api-access-6bvfp\") pod \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\" (UID: \"d1a7bbce-13c8-4622-8af2-8eae3bad4f78\") "
Nov 03 20:31:50.277072 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.277031    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-openshift-ca" pod "d1a7bbce-13c8-4622-8af2-8eae3bad4f78" (UID: "d1a7bbce-13c8-4622-8af2-8eae3bad4f78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:50.277476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.277448    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.280315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.280285    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-xks-ca" pod "d1a7bbce-13c8-4622-8af2-8eae3bad4f78" (UID: "d1a7bbce-13c8-4622-8af2-8eae3bad4f78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:50.281262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.281237    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-kube-api-access-6bvfp" pod "d1a7bbce-13c8-4622-8af2-8eae3bad4f78" (UID: "d1a7bbce-13c8-4622-8af2-8eae3bad4f78"). InnerVolumeSpecName "kube-api-access-6bvfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:50.379989 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.379884    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.379989 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.379933    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bvfp\" (UniqueName: \"kubernetes.io/projected/d1a7bbce-13c8-4622-8af2-8eae3bad4f78-kube-api-access-6bvfp\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.540180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.540140    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:50.872118 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.872016    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"
Nov 03 20:31:50.872118 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.872041    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2" event={"ID":"d1a7bbce-13c8-4622-8af2-8eae3bad4f78","Type":"ContainerDied","Data":"12fad0a06b96247039c3d9b936a77e2e72c6fefdfd28b4a403e58734cddc7ffb"}
Nov 03 20:31:50.872118 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.872111    5659 scope.go:122] "RemoveContainer" containerID="119d43a72646c5af498121c1d0cdc2e0874050522d72f808284cf5aae7e45cbc"
Nov 03 20:31:50.877821 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.877768    5659 generic.go:356] "Generic (PLEG): container finished" podID="025ae598-4df8-4574-a9b2-677c19a15fb7" containerID="156914f3a1099c6d26fe0a7599ea849db0162ed73540e3d596e7f7761f59eb21" exitCode=22
Nov 03 20:31:50.878287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.877947    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk" event={"ID":"025ae598-4df8-4574-a9b2-677c19a15fb7","Type":"ContainerDied","Data":"156914f3a1099c6d26fe0a7599ea849db0162ed73540e3d596e7f7761f59eb21"}
Nov 03 20:31:50.880513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.878523    5659 scope.go:122] "RemoveContainer" containerID="156914f3a1099c6d26fe0a7599ea849db0162ed73540e3d596e7f7761f59eb21"
Nov 03 20:31:50.880513 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:50.878976    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-jvbjk_odh-ai-gateway-infra(025ae598-4df8-4574-a9b2-677c19a15fb7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk" podUID="025ae598-4df8-4574-a9b2-677c19a15fb7"
Nov 03 20:31:50.882109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.882087    5659 generic.go:356] "Generic (PLEG): container finished" podID="6fde0783-085d-44fe-a55e-18700eaa6e88" containerID="1a1707c2faecd88955e272ad167547d91353950a323a5c49e29d2072086ceac6" exitCode=22
Nov 03 20:31:50.882220 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.882149    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw" event={"ID":"6fde0783-085d-44fe-a55e-18700eaa6e88","Type":"ContainerDied","Data":"1a1707c2faecd88955e272ad167547d91353950a323a5c49e29d2072086ceac6"}
Nov 03 20:31:50.882543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.882511    5659 scope.go:122] "RemoveContainer" containerID="1a1707c2faecd88955e272ad167547d91353950a323a5c49e29d2072086ceac6"
Nov 03 20:31:50.883017 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:50.882940    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-s5pjw_odh-ai-gateway-infra(6fde0783-085d-44fe-a55e-18700eaa6e88)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88"
Nov 03 20:31:50.888063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.888044    5659 scope.go:122] "RemoveContainer" containerID="fcab96818331d1fdb9c070623f405bccf47d5be211ebc4306c82fcdd5786668b"
Nov 03 20:31:50.905705 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.905657    5659 scope.go:122] "RemoveContainer" containerID="3382b54b90aa0b23f8d983a4707961ee76aba0b2efe07ce30b559039626b7c09"
Nov 03 20:31:50.963365 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.960148    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"]
Nov 03 20:31:50.963365 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.960225    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-jnzr2"]
Nov 03 20:31:50.963365 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:50.960285    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"]
Nov 03 20:31:51.221069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:51.221036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:51.895628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:51.895589    5659 scope.go:122] "RemoveContainer" containerID="1a1707c2faecd88955e272ad167547d91353950a323a5c49e29d2072086ceac6"
Nov 03 20:31:51.896235 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:51.896197    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-s5pjw_odh-ai-gateway-infra(6fde0783-085d-44fe-a55e-18700eaa6e88)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88"
Nov 03 20:31:51.997495 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:51.997444    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78" path="/var/lib/kubelet/pods/d1a7bbce-13c8-4622-8af2-8eae3bad4f78/volumes"
Nov 03 20:31:51.998149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:51.998124    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" path="/var/lib/kubelet/pods/d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0/volumes"
Nov 03 20:31:52.058344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.058313    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:52.170658 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.170625    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"
Nov 03 20:31:52.249568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.249529    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-xks-ca\") pod \"025ae598-4df8-4574-a9b2-677c19a15fb7\" (UID: \"025ae598-4df8-4574-a9b2-677c19a15fb7\") "
Nov 03 20:31:52.249784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.249758    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025ae598-4df8-4574-a9b2-677c19a15fb7-kube-api-access-ks94q\" (UniqueName: \"kubernetes.io/projected/025ae598-4df8-4574-a9b2-677c19a15fb7-kube-api-access-ks94q\") pod \"025ae598-4df8-4574-a9b2-677c19a15fb7\" (UID: \"025ae598-4df8-4574-a9b2-677c19a15fb7\") "
Nov 03 20:31:52.250019 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.249995    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-openshift-ca\") pod \"025ae598-4df8-4574-a9b2-677c19a15fb7\" (UID: \"025ae598-4df8-4574-a9b2-677c19a15fb7\") "
Nov 03 20:31:52.250406 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.250377    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-openshift-ca" pod "025ae598-4df8-4574-a9b2-677c19a15fb7" (UID: "025ae598-4df8-4574-a9b2-677c19a15fb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:52.251350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.251323    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:52.252501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.252459    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-xks-ca" pod "025ae598-4df8-4574-a9b2-677c19a15fb7" (UID: "025ae598-4df8-4574-a9b2-677c19a15fb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:52.253777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.253750    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025ae598-4df8-4574-a9b2-677c19a15fb7-kube-api-access-ks94q" pod "025ae598-4df8-4574-a9b2-677c19a15fb7" (UID: "025ae598-4df8-4574-a9b2-677c19a15fb7"). InnerVolumeSpecName "kube-api-access-ks94q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:52.352925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.352885    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/025ae598-4df8-4574-a9b2-677c19a15fb7-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:52.352925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.352919    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks94q\" (UniqueName: \"kubernetes.io/projected/025ae598-4df8-4574-a9b2-677c19a15fb7-kube-api-access-ks94q\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:52.673461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.673422    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"]
Nov 03 20:31:52.674351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674326    5659 cpu_manager.go:414] "containerMap: removing container" podUID="025ae598-4df8-4574-a9b2-677c19a15fb7" containerName="revoke-keys"
Nov 03 20:31:52.674351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674351    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025ae598-4df8-4574-a9b2-677c19a15fb7" containerName="revoke-keys"
Nov 03 20:31:52.674584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674376    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" containerName="revoke-keys"
Nov 03 20:31:52.674584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674388    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" containerName="revoke-keys"
Nov 03 20:31:52.674584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674399    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78" containerName="revoke-keys"
Nov 03 20:31:52.674584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674406    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78" containerName="revoke-keys"
Nov 03 20:31:52.674584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674428    5659 cpu_manager.go:414] "containerMap: removing container" podUID="025ae598-4df8-4574-a9b2-677c19a15fb7" containerName="revoke-keys"
Nov 03 20:31:52.674584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674435    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025ae598-4df8-4574-a9b2-677c19a15fb7" containerName="revoke-keys"
Nov 03 20:31:52.674584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674458    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78" containerName="revoke-keys"
Nov 03 20:31:52.674584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674465    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78" containerName="revoke-keys"
Nov 03 20:31:52.674584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674498    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" containerName="revoke-keys"
Nov 03 20:31:52.674584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674505    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" containerName="revoke-keys"
Nov 03 20:31:52.674584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674526    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" containerName="revoke-keys"
Nov 03 20:31:52.674584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674533    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" containerName="revoke-keys"
Nov 03 20:31:52.675209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674775    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" containerName="revoke-keys"
Nov 03 20:31:52.675209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674789    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78" containerName="revoke-keys"
Nov 03 20:31:52.675209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674801    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="025ae598-4df8-4574-a9b2-677c19a15fb7" containerName="revoke-keys"
Nov 03 20:31:52.675209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674814    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" containerName="revoke-keys"
Nov 03 20:31:52.675209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674826    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78" containerName="revoke-keys"
Nov 03 20:31:52.675209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.674846    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="025ae598-4df8-4574-a9b2-677c19a15fb7" containerName="revoke-keys"
Nov 03 20:31:52.679588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.679538    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"
Nov 03 20:31:52.686695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.686667    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"]
Nov 03 20:31:52.760461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.760412    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xzsd\" (UniqueName: \"kubernetes.io/projected/bc8b2605-2520-455c-aa28-6d81b5601ab8-kube-api-access-6xzsd\") pod \"maas-api-revoke-sub-f14db2bd-thg6c\" (UID: \"bc8b2605-2520-455c-aa28-6d81b5601ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"
Nov 03 20:31:52.760636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.760582    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-thg6c\" (UID: \"bc8b2605-2520-455c-aa28-6d81b5601ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"
Nov 03 20:31:52.760736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.760707    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-thg6c\" (UID: \"bc8b2605-2520-455c-aa28-6d81b5601ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"
Nov 03 20:31:52.846353 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.846320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:52.874507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.868896    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-thg6c\" (UID: \"bc8b2605-2520-455c-aa28-6d81b5601ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"
Nov 03 20:31:52.874507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.869208    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xzsd\" (UniqueName: \"kubernetes.io/projected/bc8b2605-2520-455c-aa28-6d81b5601ab8-kube-api-access-6xzsd\") pod \"maas-api-revoke-sub-f14db2bd-thg6c\" (UID: \"bc8b2605-2520-455c-aa28-6d81b5601ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"
Nov 03 20:31:52.874507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.869350    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-thg6c\" (UID: \"bc8b2605-2520-455c-aa28-6d81b5601ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"
Nov 03 20:31:52.874507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.870711    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-thg6c\" (UID: \"bc8b2605-2520-455c-aa28-6d81b5601ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"
Nov 03 20:31:52.898505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.890037    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-thg6c\" (UID: \"bc8b2605-2520-455c-aa28-6d81b5601ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"
Nov 03 20:31:52.928405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.903815    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xzsd\" (UniqueName: \"kubernetes.io/projected/bc8b2605-2520-455c-aa28-6d81b5601ab8-kube-api-access-6xzsd\") pod \"maas-api-revoke-sub-f14db2bd-thg6c\" (UID: \"bc8b2605-2520-455c-aa28-6d81b5601ab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"
Nov 03 20:31:52.935132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.935076    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk" event={"ID":"025ae598-4df8-4574-a9b2-677c19a15fb7","Type":"ContainerDied","Data":"83dd1fc66e7874c37d745d9dbec9b37b628a4ee7259cc6ef4efb7e0960f7882c"}
Nov 03 20:31:52.935338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.935188    5659 scope.go:122] "RemoveContainer" containerID="156914f3a1099c6d26fe0a7599ea849db0162ed73540e3d596e7f7761f59eb21"
Nov 03 20:31:52.935412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.935385    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"
Nov 03 20:31:52.964738 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.964709    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"]
Nov 03 20:31:52.969671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:52.969638    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-jvbjk"]
Nov 03 20:31:53.000947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:53.000914    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"
Nov 03 20:31:53.293353 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:31:53.293316    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8b2605_2520_455c_aa28_6d81b5601ab8.slice/crio-b6d9455661da9d8ace67c77f62533819f3aa88b913fe85091fc66711469fb45a WatchSource:0}: Error finding container b6d9455661da9d8ace67c77f62533819f3aa88b913fe85091fc66711469fb45a: Status 404 returned error can't find the container with id b6d9455661da9d8ace67c77f62533819f3aa88b913fe85091fc66711469fb45a
Nov 03 20:31:53.295051 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:53.295025    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"]
Nov 03 20:31:53.638718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:53.638681    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:53.945922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:53.945824    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c" event={"ID":"bc8b2605-2520-455c-aa28-6d81b5601ab8","Type":"ContainerStarted","Data":"7544b285e1a3d8b629555a8c31af97bb80718481265bfab5189ae8d1d692719b"}
Nov 03 20:31:53.945922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:53.945885    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c" event={"ID":"bc8b2605-2520-455c-aa28-6d81b5601ab8","Type":"ContainerStarted","Data":"b6d9455661da9d8ace67c77f62533819f3aa88b913fe85091fc66711469fb45a"}
Nov 03 20:31:53.969321 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:53.969261    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c" podStartSLOduration=1.9692442030000001 podStartE2EDuration="1.969244203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:53.964128727 +0000 UTC m=+3266.119031146" watchObservedRunningTime="2026-11-03 20:31:53.969244203 +0000 UTC m=+3266.124146623"
Nov 03 20:31:53.975456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:53.975421    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025ae598-4df8-4574-a9b2-677c19a15fb7" path="/var/lib/kubelet/pods/025ae598-4df8-4574-a9b2-677c19a15fb7/volumes"
Nov 03 20:31:54.526038 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:54.525995    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:54.966551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:54.964989    5659 generic.go:356] "Generic (PLEG): container finished" podID="bc8b2605-2520-455c-aa28-6d81b5601ab8" containerID="7544b285e1a3d8b629555a8c31af97bb80718481265bfab5189ae8d1d692719b" exitCode=22
Nov 03 20:31:54.966551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:54.965211    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c" event={"ID":"bc8b2605-2520-455c-aa28-6d81b5601ab8","Type":"ContainerDied","Data":"7544b285e1a3d8b629555a8c31af97bb80718481265bfab5189ae8d1d692719b"}
Nov 03 20:31:54.966551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:54.965640    5659 scope.go:122] "RemoveContainer" containerID="7544b285e1a3d8b629555a8c31af97bb80718481265bfab5189ae8d1d692719b"
Nov 03 20:31:55.439532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:55.439453    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:55.974808 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:55.974769    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"]
Nov 03 20:31:55.976958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:55.976046    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78" containerName="revoke-keys"
Nov 03 20:31:55.976958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:55.976072    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78" containerName="revoke-keys"
Nov 03 20:31:55.976958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:55.976099    5659 cpu_manager.go:414] "containerMap: removing container" podUID="025ae598-4df8-4574-a9b2-677c19a15fb7" containerName="revoke-keys"
Nov 03 20:31:55.976958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:55.976107    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025ae598-4df8-4574-a9b2-677c19a15fb7" containerName="revoke-keys"
Nov 03 20:31:55.976958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:55.976299    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63ae334-c34a-4f9f-b7c8-3c82a0cb84b0" containerName="revoke-keys"
Nov 03 20:31:55.976958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:55.976315    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="025ae598-4df8-4574-a9b2-677c19a15fb7" containerName="revoke-keys"
Nov 03 20:31:55.976958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:55.976333    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a7bbce-13c8-4622-8af2-8eae3bad4f78" containerName="revoke-keys"
Nov 03 20:31:55.982208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:55.982182    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c" event={"ID":"bc8b2605-2520-455c-aa28-6d81b5601ab8","Type":"ContainerStarted","Data":"609da6cab4c5867461872e8c9b30b9ee63f8837e5145853d7f887333604b0075"}
Nov 03 20:31:55.982329 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:55.982270    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"
Nov 03 20:31:55.983782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:55.983756    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"]
Nov 03 20:31:56.091797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.091768    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m54r\" (UniqueName: \"kubernetes.io/projected/66914ad7-0ead-4525-8906-537ed24eda95-kube-api-access-2m54r\") pod \"maas-api-revoke-sub-4bdd0b59-286lt\" (UID: \"66914ad7-0ead-4525-8906-537ed24eda95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"
Nov 03 20:31:56.091935 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.091812    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdd0b59-286lt\" (UID: \"66914ad7-0ead-4525-8906-537ed24eda95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"
Nov 03 20:31:56.092071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.092046    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdd0b59-286lt\" (UID: \"66914ad7-0ead-4525-8906-537ed24eda95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"
Nov 03 20:31:56.193988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.193957    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m54r\" (UniqueName: \"kubernetes.io/projected/66914ad7-0ead-4525-8906-537ed24eda95-kube-api-access-2m54r\") pod \"maas-api-revoke-sub-4bdd0b59-286lt\" (UID: \"66914ad7-0ead-4525-8906-537ed24eda95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"
Nov 03 20:31:56.194142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.193996    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdd0b59-286lt\" (UID: \"66914ad7-0ead-4525-8906-537ed24eda95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"
Nov 03 20:31:56.194142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.194106    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdd0b59-286lt\" (UID: \"66914ad7-0ead-4525-8906-537ed24eda95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"
Nov 03 20:31:56.194790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.194764    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bdd0b59-286lt\" (UID: \"66914ad7-0ead-4525-8906-537ed24eda95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"
Nov 03 20:31:56.196375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.196351    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bdd0b59-286lt\" (UID: \"66914ad7-0ead-4525-8906-537ed24eda95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"
Nov 03 20:31:56.209057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.209033    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m54r\" (UniqueName: \"kubernetes.io/projected/66914ad7-0ead-4525-8906-537ed24eda95-kube-api-access-2m54r\") pod \"maas-api-revoke-sub-4bdd0b59-286lt\" (UID: \"66914ad7-0ead-4525-8906-537ed24eda95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"
Nov 03 20:31:56.304094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.304019    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"
Nov 03 20:31:56.330818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.330780    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:56.707955 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:31:56.707924    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66914ad7_0ead_4525_8906_537ed24eda95.slice/crio-344a20779d2359c07eb95f7e6ab791deb00b107a90e048b92cc94e04d20c5359 WatchSource:0}: Error finding container 344a20779d2359c07eb95f7e6ab791deb00b107a90e048b92cc94e04d20c5359: Status 404 returned error can't find the container with id 344a20779d2359c07eb95f7e6ab791deb00b107a90e048b92cc94e04d20c5359
Nov 03 20:31:56.708421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.708398    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"]
Nov 03 20:31:56.990411 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.990327    5659 generic.go:356] "Generic (PLEG): container finished" podID="bc8b2605-2520-455c-aa28-6d81b5601ab8" containerID="609da6cab4c5867461872e8c9b30b9ee63f8837e5145853d7f887333604b0075" exitCode=22
Nov 03 20:31:56.990970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.990416    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c" event={"ID":"bc8b2605-2520-455c-aa28-6d81b5601ab8","Type":"ContainerDied","Data":"609da6cab4c5867461872e8c9b30b9ee63f8837e5145853d7f887333604b0075"}
Nov 03 20:31:56.990970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.990530    5659 scope.go:122] "RemoveContainer" containerID="7544b285e1a3d8b629555a8c31af97bb80718481265bfab5189ae8d1d692719b"
Nov 03 20:31:56.990970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.990823    5659 scope.go:122] "RemoveContainer" containerID="609da6cab4c5867461872e8c9b30b9ee63f8837e5145853d7f887333604b0075"
Nov 03 20:31:56.991344 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:56.991321    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-thg6c_odh-ai-gateway-infra(bc8b2605-2520-455c-aa28-6d81b5601ab8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c" podUID="bc8b2605-2520-455c-aa28-6d81b5601ab8"
Nov 03 20:31:56.992126 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.992098    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:56.992321 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:56.992301    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt" event={"ID":"66914ad7-0ead-4525-8906-537ed24eda95","Type":"ContainerStarted","Data":"344a20779d2359c07eb95f7e6ab791deb00b107a90e048b92cc94e04d20c5359"}
Nov 03 20:31:57.218081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:57.218043    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:57.343555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:57.341370    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:57.972504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:57.972390    5659 scope.go:122] "RemoveContainer" containerID="35a9dd8133650ea152120cdf310b246b72f7f037b87810be66950247db0b0772"
Nov 03 20:31:57.979215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:57.974170    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:58.038360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:58.037507    5659 scope.go:122] "RemoveContainer" containerID="609da6cab4c5867461872e8c9b30b9ee63f8837e5145853d7f887333604b0075"
Nov 03 20:31:58.038360 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:31:58.038009    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-thg6c_odh-ai-gateway-infra(bc8b2605-2520-455c-aa28-6d81b5601ab8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c" podUID="bc8b2605-2520-455c-aa28-6d81b5601ab8"
Nov 03 20:31:58.051252 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:58.049805    5659 generic.go:356] "Generic (PLEG): container finished" podID="66914ad7-0ead-4525-8906-537ed24eda95" containerID="2cd59ad1ba5aa6aad4c5a6c4dc89d433e4c762ac26587ebfc5b460752007b6c9" exitCode=22
Nov 03 20:31:58.051252 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:58.049870    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt" event={"ID":"66914ad7-0ead-4525-8906-537ed24eda95","Type":"ContainerDied","Data":"2cd59ad1ba5aa6aad4c5a6c4dc89d433e4c762ac26587ebfc5b460752007b6c9"}
Nov 03 20:31:58.051252 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:58.050461    5659 scope.go:122] "RemoveContainer" containerID="2cd59ad1ba5aa6aad4c5a6c4dc89d433e4c762ac26587ebfc5b460752007b6c9"
Nov 03 20:31:58.658351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:58.658320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:58.782232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:58.782056    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"]
Nov 03 20:31:58.790201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:58.790173    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"]
Nov 03 20:31:58.790362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:58.790302    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"
Nov 03 20:31:58.925901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:58.925870    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-vvgj6\" (UID: \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"
Nov 03 20:31:58.926176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:58.926159    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-vvgj6\" (UID: \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"
Nov 03 20:31:58.926223 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:58.926193    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dctlq\" (UniqueName: \"kubernetes.io/projected/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-kube-api-access-dctlq\") pod \"maas-api-revoke-sub-fb827799-vvgj6\" (UID: \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"
Nov 03 20:31:58.962102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:58.962065    5659 scope.go:122] "RemoveContainer" containerID="a0afbe7257d8cc4cc6bf34438d38ad1235aa1fc9ac298f440219e318e1104251"
Nov 03 20:31:59.027206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:59.027171    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-vvgj6\" (UID: \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"
Nov 03 20:31:59.027444 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:59.027425    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-vvgj6\" (UID: \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"
Nov 03 20:31:59.027553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:59.027461    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dctlq\" (UniqueName: \"kubernetes.io/projected/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-kube-api-access-dctlq\") pod \"maas-api-revoke-sub-fb827799-vvgj6\" (UID: \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"
Nov 03 20:31:59.028076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:59.028055    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-vvgj6\" (UID: \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"
Nov 03 20:31:59.029615 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:59.029596    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-vvgj6\" (UID: \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"
Nov 03 20:31:59.039819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:59.039763    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dctlq\" (UniqueName: \"kubernetes.io/projected/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-kube-api-access-dctlq\") pod \"maas-api-revoke-sub-fb827799-vvgj6\" (UID: \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"
Nov 03 20:31:59.061350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:59.061319    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv" event={"ID":"b1050e12-dfb8-41df-876f-c754f3567fc5","Type":"ContainerStarted","Data":"2ee8b61d526ada6cecf9191792702edd1f2d1516561350fc7dc4d5746a2890d6"}
Nov 03 20:31:59.063801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:59.063776    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt" event={"ID":"66914ad7-0ead-4525-8906-537ed24eda95","Type":"ContainerStarted","Data":"46803512e5fdca6f59bd1a341348adc3e071097e2c1d345a67663226f2e58680"}
Nov 03 20:31:59.098059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:59.097997    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt" podStartSLOduration=4.097983186 podStartE2EDuration="4.097983186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:59.096382904 +0000 UTC m=+3271.251285334" watchObservedRunningTime="2026-11-03 20:31:59.097983186 +0000 UTC m=+3271.252885604"
Nov 03 20:31:59.116082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:59.116054    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"
Nov 03 20:31:59.417646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:59.417613    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"]
Nov 03 20:31:59.419422 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:31:59.419388    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d6fc0d_cefc_4d63_bc06_70e3b394ab20.slice/crio-6dbbd7a4577a2664ff2ef49159687bcb11787fc82907c31789a866eef704a95f WatchSource:0}: Error finding container 6dbbd7a4577a2664ff2ef49159687bcb11787fc82907c31789a866eef704a95f: Status 404 returned error can't find the container with id 6dbbd7a4577a2664ff2ef49159687bcb11787fc82907c31789a866eef704a95f
Nov 03 20:31:59.427626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:31:59.427605    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:00.075444 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.075411    5659 generic.go:356] "Generic (PLEG): container finished" podID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" containerID="547c0e1419756ff1883c81da9bc6a50f3e36139299df2bd20f8e8533aa5d338c" exitCode=22
Nov 03 20:32:00.076209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.075848    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb" event={"ID":"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef","Type":"ContainerDied","Data":"547c0e1419756ff1883c81da9bc6a50f3e36139299df2bd20f8e8533aa5d338c"}
Nov 03 20:32:00.076209 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.075917    5659 scope.go:122] "RemoveContainer" containerID="a0afbe7257d8cc4cc6bf34438d38ad1235aa1fc9ac298f440219e318e1104251"
Nov 03 20:32:00.076577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.076554    5659 scope.go:122] "RemoveContainer" containerID="547c0e1419756ff1883c81da9bc6a50f3e36139299df2bd20f8e8533aa5d338c"
Nov 03 20:32:00.077058 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:00.077028    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-ccfxb_odh-ai-gateway-infra(edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb" podUID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef"
Nov 03 20:32:00.078876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.078851    5659 generic.go:356] "Generic (PLEG): container finished" podID="66914ad7-0ead-4525-8906-537ed24eda95" containerID="46803512e5fdca6f59bd1a341348adc3e071097e2c1d345a67663226f2e58680" exitCode=22
Nov 03 20:32:00.078962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.078944    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt" event={"ID":"66914ad7-0ead-4525-8906-537ed24eda95","Type":"ContainerDied","Data":"46803512e5fdca6f59bd1a341348adc3e071097e2c1d345a67663226f2e58680"}
Nov 03 20:32:00.079220 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.079198    5659 scope.go:122] "RemoveContainer" containerID="46803512e5fdca6f59bd1a341348adc3e071097e2c1d345a67663226f2e58680"
Nov 03 20:32:00.079581 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:00.079554    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-4bdd0b59-286lt_odh-ai-gateway-infra(66914ad7-0ead-4525-8906-537ed24eda95)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt" podUID="66914ad7-0ead-4525-8906-537ed24eda95"
Nov 03 20:32:00.080861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.080833    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6" event={"ID":"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20","Type":"ContainerStarted","Data":"fb41165d8744f9e248d31d5503f0e29eae8ab55dc211e6070d3971f20330171f"}
Nov 03 20:32:00.080973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.080887    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6" event={"ID":"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20","Type":"ContainerStarted","Data":"6dbbd7a4577a2664ff2ef49159687bcb11787fc82907c31789a866eef704a95f"}
Nov 03 20:32:00.082871 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.082847    5659 generic.go:356] "Generic (PLEG): container finished" podID="b1050e12-dfb8-41df-876f-c754f3567fc5" containerID="2ee8b61d526ada6cecf9191792702edd1f2d1516561350fc7dc4d5746a2890d6" exitCode=22
Nov 03 20:32:00.082952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.082891    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv" event={"ID":"b1050e12-dfb8-41df-876f-c754f3567fc5","Type":"ContainerDied","Data":"2ee8b61d526ada6cecf9191792702edd1f2d1516561350fc7dc4d5746a2890d6"}
Nov 03 20:32:00.083402 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.083384    5659 scope.go:122] "RemoveContainer" containerID="2ee8b61d526ada6cecf9191792702edd1f2d1516561350fc7dc4d5746a2890d6"
Nov 03 20:32:00.083751 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:00.083728    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-tg7tv_odh-ai-gateway-infra(b1050e12-dfb8-41df-876f-c754f3567fc5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv" podUID="b1050e12-dfb8-41df-876f-c754f3567fc5"
Nov 03 20:32:00.092953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.092928    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"]
Nov 03 20:32:00.093454 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.093437    5659 scope.go:122] "RemoveContainer" containerID="2cd59ad1ba5aa6aad4c5a6c4dc89d433e4c762ac26587ebfc5b460752007b6c9"
Nov 03 20:32:00.110413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.110394    5659 scope.go:122] "RemoveContainer" containerID="35a9dd8133650ea152120cdf310b246b72f7f037b87810be66950247db0b0772"
Nov 03 20:32:00.125956 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.125936    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:00.224985 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.222571    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6" podStartSLOduration=2.222545043 podStartE2EDuration="2.222545043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:00.198848077 +0000 UTC m=+3272.353750498" watchObservedRunningTime="2026-11-03 20:32:00.222545043 +0000 UTC m=+3272.377447462"
Nov 03 20:32:00.850547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:00.850509    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:01.095953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.095924    5659 scope.go:122] "RemoveContainer" containerID="46803512e5fdca6f59bd1a341348adc3e071097e2c1d345a67663226f2e58680"
Nov 03 20:32:01.096421 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:01.096378    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-4bdd0b59-286lt_odh-ai-gateway-infra(66914ad7-0ead-4525-8906-537ed24eda95)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt" podUID="66914ad7-0ead-4525-8906-537ed24eda95"
Nov 03 20:32:01.097288 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.097265    5659 generic.go:356] "Generic (PLEG): container finished" podID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" containerID="fb41165d8744f9e248d31d5503f0e29eae8ab55dc211e6070d3971f20330171f" exitCode=7
Nov 03 20:32:01.097358 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.097329    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6" event={"ID":"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20","Type":"ContainerDied","Data":"fb41165d8744f9e248d31d5503f0e29eae8ab55dc211e6070d3971f20330171f"}
Nov 03 20:32:01.097811 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.097645    5659 scope.go:122] "RemoveContainer" containerID="fb41165d8744f9e248d31d5503f0e29eae8ab55dc211e6070d3971f20330171f"
Nov 03 20:32:01.140677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.140630    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"]
Nov 03 20:32:01.390907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.390875    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"
Nov 03 20:32:01.391255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.391227    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"]
Nov 03 20:32:01.392752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.392726    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b1050e12-dfb8-41df-876f-c754f3567fc5" containerName="revoke-keys"
Nov 03 20:32:01.392752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.392752    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1050e12-dfb8-41df-876f-c754f3567fc5" containerName="revoke-keys"
Nov 03 20:32:01.392926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.392786    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b1050e12-dfb8-41df-876f-c754f3567fc5" containerName="revoke-keys"
Nov 03 20:32:01.392926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.392795    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1050e12-dfb8-41df-876f-c754f3567fc5" containerName="revoke-keys"
Nov 03 20:32:01.392926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.392811    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b1050e12-dfb8-41df-876f-c754f3567fc5" containerName="revoke-keys"
Nov 03 20:32:01.392926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.392819    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1050e12-dfb8-41df-876f-c754f3567fc5" containerName="revoke-keys"
Nov 03 20:32:01.393135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.393012    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1050e12-dfb8-41df-876f-c754f3567fc5" containerName="revoke-keys"
Nov 03 20:32:01.393135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.393028    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1050e12-dfb8-41df-876f-c754f3567fc5" containerName="revoke-keys"
Nov 03 20:32:01.393135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.393042    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1050e12-dfb8-41df-876f-c754f3567fc5" containerName="revoke-keys"
Nov 03 20:32:01.399548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.399506    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"]
Nov 03 20:32:01.399686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.399609    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"
Nov 03 20:32:01.450404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.450369    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1050e12-dfb8-41df-876f-c754f3567fc5-kube-api-access-vr4fs\" (UniqueName: \"kubernetes.io/projected/b1050e12-dfb8-41df-876f-c754f3567fc5-kube-api-access-vr4fs\") pod \"b1050e12-dfb8-41df-876f-c754f3567fc5\" (UID: \"b1050e12-dfb8-41df-876f-c754f3567fc5\") "
Nov 03 20:32:01.451132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.451107    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-xks-ca\") pod \"b1050e12-dfb8-41df-876f-c754f3567fc5\" (UID: \"b1050e12-dfb8-41df-876f-c754f3567fc5\") "
Nov 03 20:32:01.451614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.451346    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-openshift-ca\") pod \"b1050e12-dfb8-41df-876f-c754f3567fc5\" (UID: \"b1050e12-dfb8-41df-876f-c754f3567fc5\") "
Nov 03 20:32:01.451887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.451853    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5n7\" (UniqueName: \"kubernetes.io/projected/f4ecbc99-1b53-4a25-8944-20f24f9a4044-kube-api-access-2p5n7\") pod \"maas-api-revoke-sub-403cf98d-r99x5\" (UID: \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"
Nov 03 20:32:01.451993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.451975    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-r99x5\" (UID: \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"
Nov 03 20:32:01.452062 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.452021    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-r99x5\" (UID: \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"
Nov 03 20:32:01.452676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.452653    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-openshift-ca" pod "b1050e12-dfb8-41df-876f-c754f3567fc5" (UID: "b1050e12-dfb8-41df-876f-c754f3567fc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:01.455936 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.455909    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1050e12-dfb8-41df-876f-c754f3567fc5-kube-api-access-vr4fs" pod "b1050e12-dfb8-41df-876f-c754f3567fc5" (UID: "b1050e12-dfb8-41df-876f-c754f3567fc5"). InnerVolumeSpecName "kube-api-access-vr4fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:01.457245 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.457221    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-xks-ca" pod "b1050e12-dfb8-41df-876f-c754f3567fc5" (UID: "b1050e12-dfb8-41df-876f-c754f3567fc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:01.501475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.501445    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"
Nov 03 20:32:01.553596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.553555    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-kube-api-access-nhxb5\" (UniqueName: \"kubernetes.io/projected/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-kube-api-access-nhxb5\") pod \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\" (UID: \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\") "
Nov 03 20:32:01.553867 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.553638    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-xks-ca\") pod \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\" (UID: \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\") "
Nov 03 20:32:01.553867 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.553680    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-openshift-ca\") pod \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\" (UID: \"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef\") "
Nov 03 20:32:01.554210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.554183    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5n7\" (UniqueName: \"kubernetes.io/projected/f4ecbc99-1b53-4a25-8944-20f24f9a4044-kube-api-access-2p5n7\") pod \"maas-api-revoke-sub-403cf98d-r99x5\" (UID: \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"
Nov 03 20:32:01.554349 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.554329    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-r99x5\" (UID: \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"
Nov 03 20:32:01.554429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.554375    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-r99x5\" (UID: \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"
Nov 03 20:32:01.554875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.554853    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr4fs\" (UniqueName: \"kubernetes.io/projected/b1050e12-dfb8-41df-876f-c754f3567fc5-kube-api-access-vr4fs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:01.555029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.554879    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:01.555029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.554894    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1050e12-dfb8-41df-876f-c754f3567fc5-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:01.555919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.555885    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-r99x5\" (UID: \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"
Nov 03 20:32:01.556078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.556050    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-openshift-ca" pod "edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" (UID: "edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:01.558125 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.558093    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-kube-api-access-nhxb5" pod "edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" (UID: "edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef"). InnerVolumeSpecName "kube-api-access-nhxb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:01.558696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.558651    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-xks-ca" pod "edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" (UID: "edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:01.558696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.558655    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-r99x5\" (UID: \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"
Nov 03 20:32:01.567473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.567439    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5n7\" (UniqueName: \"kubernetes.io/projected/f4ecbc99-1b53-4a25-8944-20f24f9a4044-kube-api-access-2p5n7\") pod \"maas-api-revoke-sub-403cf98d-r99x5\" (UID: \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"
Nov 03 20:32:01.656728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.656689    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhxb5\" (UniqueName: \"kubernetes.io/projected/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-kube-api-access-nhxb5\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:01.656728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.656730    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:01.656924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.656749    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:01.760576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.759084    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"
Nov 03 20:32:01.760576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.759934    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:01.962059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:01.962023    5659 scope.go:122] "RemoveContainer" containerID="1a1707c2faecd88955e272ad167547d91353950a323a5c49e29d2072086ceac6"
Nov 03 20:32:02.100857 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.100821    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"]
Nov 03 20:32:02.110675 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.110626    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"
Nov 03 20:32:02.110806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.110702    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb" event={"ID":"edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef","Type":"ContainerDied","Data":"0a5aa747bdc5702e9d596c852941e2a425525983b6705045dc31d695e37742f5"}
Nov 03 20:32:02.110806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.110759    5659 scope.go:122] "RemoveContainer" containerID="547c0e1419756ff1883c81da9bc6a50f3e36139299df2bd20f8e8533aa5d338c"
Nov 03 20:32:02.114899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.114871    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6" event={"ID":"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20","Type":"ContainerStarted","Data":"f0abcc9f6027c84f772ede402f068a7ca290e16e3ed2ad866c80cff3249af30a"}
Nov 03 20:32:02.117732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.117714    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"
Nov 03 20:32:02.117836 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.117749    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv" event={"ID":"b1050e12-dfb8-41df-876f-c754f3567fc5","Type":"ContainerDied","Data":"0576500aabb73b819720e1a359553ad01cbeb7de154bbf34f190f48d61c48674"}
Nov 03 20:32:02.143756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.143730    5659 scope.go:122] "RemoveContainer" containerID="2ee8b61d526ada6cecf9191792702edd1f2d1516561350fc7dc4d5746a2890d6"
Nov 03 20:32:02.158710 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.158678    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"]
Nov 03 20:32:02.172819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.172780    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-ccfxb"]
Nov 03 20:32:02.184641 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.184610    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"]
Nov 03 20:32:02.190382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.190351    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-tg7tv"]
Nov 03 20:32:02.285653 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.285621    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"]
Nov 03 20:32:02.286462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.286446    5659 cpu_manager.go:414] "containerMap: removing container" podUID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" containerName="revoke-keys"
Nov 03 20:32:02.286548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.286464    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" containerName="revoke-keys"
Nov 03 20:32:02.286548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.286491    5659 cpu_manager.go:414] "containerMap: removing container" podUID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" containerName="revoke-keys"
Nov 03 20:32:02.286548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.286497    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" containerName="revoke-keys"
Nov 03 20:32:02.286656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.286649    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" containerName="revoke-keys"
Nov 03 20:32:02.286700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.286667    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" containerName="revoke-keys"
Nov 03 20:32:02.290837 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.290818    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"
Nov 03 20:32:02.300273 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.300193    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"]
Nov 03 20:32:02.375091 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.374993    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-z8t7z\" (UID: \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"
Nov 03 20:32:02.375380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.375163    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-z8t7z\" (UID: \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"
Nov 03 20:32:02.377583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.375695    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qld6c\" (UniqueName: \"kubernetes.io/projected/48c9c490-efc4-4157-830b-8c3bb6bee5ce-kube-api-access-qld6c\") pod \"maas-api-revoke-sub-abad4031-z8t7z\" (UID: \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"
Nov 03 20:32:02.477797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.477758    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qld6c\" (UniqueName: \"kubernetes.io/projected/48c9c490-efc4-4157-830b-8c3bb6bee5ce-kube-api-access-qld6c\") pod \"maas-api-revoke-sub-abad4031-z8t7z\" (UID: \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"
Nov 03 20:32:02.477971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.477906    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-z8t7z\" (UID: \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"
Nov 03 20:32:02.477971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.477944    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-z8t7z\" (UID: \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"
Nov 03 20:32:02.479037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.479008    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-z8t7z\" (UID: \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"
Nov 03 20:32:02.480517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.480472    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-z8t7z\" (UID: \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"
Nov 03 20:32:02.508518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.508065    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"]
Nov 03 20:32:02.522542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.516114    5659 cpu_manager.go:414] "containerMap: removing container" podUID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" containerName="revoke-keys"
Nov 03 20:32:02.522542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.516146    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" containerName="revoke-keys"
Nov 03 20:32:02.522542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.516429    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" containerName="revoke-keys"
Nov 03 20:32:02.523720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.523691    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.530968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.525795    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qld6c\" (UniqueName: \"kubernetes.io/projected/48c9c490-efc4-4157-830b-8c3bb6bee5ce-kube-api-access-qld6c\") pod \"maas-api-revoke-sub-abad4031-z8t7z\" (UID: \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"
Nov 03 20:32:02.588105 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.588065    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fsb74\""
Nov 03 20:32:02.590392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.589518    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.590392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.589565    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.590392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.589668    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.590392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.589881    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.590392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.589947    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wghr\" (UniqueName: \"kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-kube-api-access-8wghr\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.590392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.589992    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.590392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.590181    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.590392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.590240    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.590392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.590267    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.596387 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.596114    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"]
Nov 03 20:32:02.609614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.608706    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"
Nov 03 20:32:02.689579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.689534    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"]
Nov 03 20:32:02.692505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.691601    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.692505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.691745    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.692505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.691804    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wghr\" (UniqueName: \"kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-kube-api-access-8wghr\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.692505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.691851    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.692505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.691888    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.692505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.691921    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.692505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.691948    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.692505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.692113    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.692505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.692142    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.693096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.692500    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.694136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.693764    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.694136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.694077    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.694136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.694129    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.696347 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.696327    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.696711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.696690    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"
Nov 03 20:32:02.696848 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.696828    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.698606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.698432    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.791659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.781002    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2b5703-dockercfg-b84j2\""
Nov 03 20:32:02.791659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.781049    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-2b5703\""
Nov 03 20:32:02.792189 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.792160    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:02.797170 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.797134    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78765763-73ba-4537-aa8a-47b8159a58c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65\" (UID: \"78765763-73ba-4537-aa8a-47b8159a58c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"
Nov 03 20:32:02.797447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.797402    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jlgl\" (UniqueName: \"kubernetes.io/projected/78765763-73ba-4537-aa8a-47b8159a58c6-kube-api-access-4jlgl\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65\" (UID: \"78765763-73ba-4537-aa8a-47b8159a58c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"
Nov 03 20:32:02.797957 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.797830    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"]
Nov 03 20:32:02.801342 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.799260    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.818045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.818018    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wghr\" (UniqueName: \"kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-kube-api-access-8wghr\") pod \"data-science-gateway-data-science-gateway-class-68d996c557ghsgt\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:02.890504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.877852    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66b59d468d-j8ndl"]
Nov 03 20:32:02.940827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:02.940536    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:03.037958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.013813    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b59d468d-j8ndl"
Nov 03 20:32:03.037958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.024134    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78765763-73ba-4537-aa8a-47b8159a58c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65\" (UID: \"78765763-73ba-4537-aa8a-47b8159a58c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"
Nov 03 20:32:03.037958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.024511    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jlgl\" (UniqueName: \"kubernetes.io/projected/78765763-73ba-4537-aa8a-47b8159a58c6-kube-api-access-4jlgl\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65\" (UID: \"78765763-73ba-4537-aa8a-47b8159a58c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"
Nov 03 20:32:03.037958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.026160    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78765763-73ba-4537-aa8a-47b8159a58c6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65\" (UID: \"78765763-73ba-4537-aa8a-47b8159a58c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"
Nov 03 20:32:03.037958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.036874    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sgdb8\""
Nov 03 20:32:03.037958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.037031    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 20:32:03.063926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.063836    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66b59d468d-j8ndl"]
Nov 03 20:32:03.069895 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.069862    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jlgl\" (UniqueName: \"kubernetes.io/projected/78765763-73ba-4537-aa8a-47b8159a58c6-kube-api-access-4jlgl\") pod \"payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65\" (UID: \"78765763-73ba-4537-aa8a-47b8159a58c6\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"
Nov 03 20:32:03.086539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.084123    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"
Nov 03 20:32:03.178506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.139876    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19730c58-ac26-42dc-bd3a-8ec0f6076264-tls-cert\") pod \"authorino-66b59d468d-j8ndl\" (UID: \"19730c58-ac26-42dc-bd3a-8ec0f6076264\") " pod="kuadrant-system/authorino-66b59d468d-j8ndl"
Nov 03 20:32:03.178506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.140377    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxnbn\" (UniqueName: \"kubernetes.io/projected/19730c58-ac26-42dc-bd3a-8ec0f6076264-kube-api-access-lxnbn\") pod \"authorino-66b59d468d-j8ndl\" (UID: \"19730c58-ac26-42dc-bd3a-8ec0f6076264\") " pod="kuadrant-system/authorino-66b59d468d-j8ndl"
Nov 03 20:32:03.178506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.163024    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"]
Nov 03 20:32:03.197981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.197923    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"]
Nov 03 20:32:03.198304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.198281    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"]
Nov 03 20:32:03.346602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.336922    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxnbn\" (UniqueName: \"kubernetes.io/projected/19730c58-ac26-42dc-bd3a-8ec0f6076264-kube-api-access-lxnbn\") pod \"authorino-66b59d468d-j8ndl\" (UID: \"19730c58-ac26-42dc-bd3a-8ec0f6076264\") " pod="kuadrant-system/authorino-66b59d468d-j8ndl"
Nov 03 20:32:03.346602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.337099    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19730c58-ac26-42dc-bd3a-8ec0f6076264-tls-cert\") pod \"authorino-66b59d468d-j8ndl\" (UID: \"19730c58-ac26-42dc-bd3a-8ec0f6076264\") " pod="kuadrant-system/authorino-66b59d468d-j8ndl"
Nov 03 20:32:03.365749 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.358438    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"]
Nov 03 20:32:03.410982 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.410941    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5" event={"ID":"f4ecbc99-1b53-4a25-8944-20f24f9a4044","Type":"ContainerStarted","Data":"92fc3947aa9412ec8c28b36caaeef30a9a374d42c3b2e377aef46c91f002f55d"}
Nov 03 20:32:03.411242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.411195    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5" event={"ID":"f4ecbc99-1b53-4a25-8944-20f24f9a4044","Type":"ContainerStarted","Data":"a623f9c94094519ce621961f28f4e3fa238c26be8eedc51c89627498ce007158"}
Nov 03 20:32:03.412102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.412081    5659 scope.go:122] "RemoveContainer" containerID="92fc3947aa9412ec8c28b36caaeef30a9a374d42c3b2e377aef46c91f002f55d"
Nov 03 20:32:03.419560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.418362    5659 generic.go:356] "Generic (PLEG): container finished" podID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" containerID="f0abcc9f6027c84f772ede402f068a7ca290e16e3ed2ad866c80cff3249af30a" exitCode=7
Nov 03 20:32:03.419560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.418441    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6" event={"ID":"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20","Type":"ContainerDied","Data":"f0abcc9f6027c84f772ede402f068a7ca290e16e3ed2ad866c80cff3249af30a"}
Nov 03 20:32:03.419560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.418511    5659 scope.go:122] "RemoveContainer" containerID="fb41165d8744f9e248d31d5503f0e29eae8ab55dc211e6070d3971f20330171f"
Nov 03 20:32:03.419560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.419003    5659 scope.go:122] "RemoveContainer" containerID="f0abcc9f6027c84f772ede402f068a7ca290e16e3ed2ad866c80cff3249af30a"
Nov 03 20:32:03.419560 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:03.419467    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-vvgj6_odh-ai-gateway-infra(c9d6fc0d-cefc-4d63-bc06-70e3b394ab20)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6" podUID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20"
Nov 03 20:32:03.439468 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.439149    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19730c58-ac26-42dc-bd3a-8ec0f6076264-tls-cert\") pod \"authorino-66b59d468d-j8ndl\" (UID: \"19730c58-ac26-42dc-bd3a-8ec0f6076264\") " pod="kuadrant-system/authorino-66b59d468d-j8ndl"
Nov 03 20:32:03.458469 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.450305    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxnbn\" (UniqueName: \"kubernetes.io/projected/19730c58-ac26-42dc-bd3a-8ec0f6076264-kube-api-access-lxnbn\") pod \"authorino-66b59d468d-j8ndl\" (UID: \"19730c58-ac26-42dc-bd3a-8ec0f6076264\") " pod="kuadrant-system/authorino-66b59d468d-j8ndl"
Nov 03 20:32:03.458469 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.456856    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw" event={"ID":"6fde0783-085d-44fe-a55e-18700eaa6e88","Type":"ContainerStarted","Data":"301180709be8957d8e6c2acfd12bed75d9e9b9f76f16a7ce8a8fb300336675c4"}
Nov 03 20:32:03.488732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.488701    5659 scope.go:122] "RemoveContainer" containerID="301180709be8957d8e6c2acfd12bed75d9e9b9f76f16a7ce8a8fb300336675c4"
Nov 03 20:32:03.489661 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:03.489634    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-s5pjw_odh-ai-gateway-infra(6fde0783-085d-44fe-a55e-18700eaa6e88)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88"
Nov 03 20:32:03.503057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.503011    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"]
Nov 03 20:32:03.508801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.503189    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy" containerID="cri-o://df64b065a07a619c7cf3705172be7c7543e8d9685ba16888bcb9054792945514" gracePeriod=30
Nov 03 20:32:03.508801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.506142    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"]
Nov 03 20:32:03.510062 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.509987    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"]
Nov 03 20:32:03.510196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.510151    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk" podUID="eb414b62-f84b-4da8-9891-b85f75868dd0" containerName="istio-proxy" containerID="cri-o://2af7ed40bd868991cc818d3c07409189994e01ea4556d31ecc0be76fc17e32a4" gracePeriod=30
Nov 03 20:32:03.522263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.521834    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"]
Nov 03 20:32:03.522263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.522190    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f" podUID="b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" containerName="istio-proxy" containerID="cri-o://5976e2e2d71acbbf7af0fd68d94b85ed4aeb64c1a30ebc81c14b5ee275d8a052" gracePeriod=30
Nov 03 20:32:03.530270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.530219    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"]
Nov 03 20:32:03.530737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.530659    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" podUID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" containerName="istio-proxy" containerID="cri-o://411a7f77513285e0bd3d10efb914958e55485b6921f9300096f36d8817aff58c" gracePeriod=30
Nov 03 20:32:03.612324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.611136    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"]
Nov 03 20:32:03.644182 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.644114    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:03.673682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.669776    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66b59d468d-j8ndl"
Nov 03 20:32:03.727411 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.727368    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"]
Nov 03 20:32:03.746900 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.746854    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:32:03.747043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.746963    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:32:03.747043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.747012    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:32:03.906947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.906780    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:03.913954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.913862    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"]
Nov 03 20:32:03.989290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.989252    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1050e12-dfb8-41df-876f-c754f3567fc5" path="/var/lib/kubelet/pods/b1050e12-dfb8-41df-876f-c754f3567fc5/volumes"
Nov 03 20:32:03.990074 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:03.990055    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef" path="/var/lib/kubelet/pods/edbbc3ca-4ab5-46af-868d-c1f3a50bf3ef/volumes"
Nov 03 20:32:04.100691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.092171    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66b59d468d-j8ndl"]
Nov 03 20:32:04.102552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.101934    5659 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:32:04.183509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.183407    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:04.346385 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.346344    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:04.485465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.485432    5659 generic.go:356] "Generic (PLEG): container finished" podID="6fde0783-085d-44fe-a55e-18700eaa6e88" containerID="301180709be8957d8e6c2acfd12bed75d9e9b9f76f16a7ce8a8fb300336675c4" exitCode=22
Nov 03 20:32:04.485703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.485513    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw" event={"ID":"6fde0783-085d-44fe-a55e-18700eaa6e88","Type":"ContainerDied","Data":"301180709be8957d8e6c2acfd12bed75d9e9b9f76f16a7ce8a8fb300336675c4"}
Nov 03 20:32:04.485703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.485620    5659 scope.go:122] "RemoveContainer" containerID="1a1707c2faecd88955e272ad167547d91353950a323a5c49e29d2072086ceac6"
Nov 03 20:32:04.486160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.486135    5659 scope.go:122] "RemoveContainer" containerID="301180709be8957d8e6c2acfd12bed75d9e9b9f76f16a7ce8a8fb300336675c4"
Nov 03 20:32:04.486924 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:04.486668    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-s5pjw_odh-ai-gateway-infra(6fde0783-085d-44fe-a55e-18700eaa6e88)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88"
Nov 03 20:32:04.487923 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.487897    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt" event={"ID":"21a70e8c-6980-4dcb-aafd-0cf5feecd588","Type":"ContainerStarted","Data":"11fb6de88e3166bb3ed3332661c84de96f809feada507d5f5678faae9959baaa"}
Nov 03 20:32:04.488016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.487948    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt" event={"ID":"21a70e8c-6980-4dcb-aafd-0cf5feecd588","Type":"ContainerStarted","Data":"4484e058c08af1933d7c05749283441abdbf143604ec572a3976621e312ddc2d"}
Nov 03 20:32:04.488016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.487948    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt" podUID="21a70e8c-6980-4dcb-aafd-0cf5feecd588" containerName="istio-proxy" containerID="cri-o://11fb6de88e3166bb3ed3332661c84de96f809feada507d5f5678faae9959baaa" gracePeriod=30
Nov 03 20:32:04.490021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.489998    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65" event={"ID":"78765763-73ba-4537-aa8a-47b8159a58c6","Type":"ContainerStarted","Data":"de87f3fa0926cd55169892f5dca7b2a6f75772887ada5b42ce946e5aa2529d3c"}
Nov 03 20:32:04.490151 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.490043    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65" event={"ID":"78765763-73ba-4537-aa8a-47b8159a58c6","Type":"ContainerStarted","Data":"3e43577cec5d14274e99eadf2a47897c1f7c0eba2a96dbfe051c76af6ead70cb"}
Nov 03 20:32:04.490287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.490223    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"
Nov 03 20:32:04.493313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.493281    5659 generic.go:356] "Generic (PLEG): container finished" podID="f4ecbc99-1b53-4a25-8944-20f24f9a4044" containerID="92fc3947aa9412ec8c28b36caaeef30a9a374d42c3b2e377aef46c91f002f55d" exitCode=22
Nov 03 20:32:04.493427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.493363    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5" event={"ID":"f4ecbc99-1b53-4a25-8944-20f24f9a4044","Type":"ContainerDied","Data":"92fc3947aa9412ec8c28b36caaeef30a9a374d42c3b2e377aef46c91f002f55d"}
Nov 03 20:32:04.493427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.493408    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5" podUID="f4ecbc99-1b53-4a25-8944-20f24f9a4044" containerName="revoke-keys" containerID="cri-o://8c1ccdc689aadd84c46877402beb5ba85fc222a2dc9e485750614e7d1f98df08" gracePeriod=30
Nov 03 20:32:04.493579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.493411    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5" event={"ID":"f4ecbc99-1b53-4a25-8944-20f24f9a4044","Type":"ContainerStarted","Data":"8c1ccdc689aadd84c46877402beb5ba85fc222a2dc9e485750614e7d1f98df08"}
Nov 03 20:32:04.495818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.495802    5659 scope.go:122] "RemoveContainer" containerID="f0abcc9f6027c84f772ede402f068a7ca290e16e3ed2ad866c80cff3249af30a"
Nov 03 20:32:04.496375 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:04.496346    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-vvgj6_odh-ai-gateway-infra(c9d6fc0d-cefc-4d63-bc06-70e3b394ab20)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6" podUID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20"
Nov 03 20:32:04.498197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.496959    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b59d468d-j8ndl" event={"ID":"19730c58-ac26-42dc-bd3a-8ec0f6076264","Type":"ContainerStarted","Data":"170650dc0856d3639b8c6738d25417d15b62b1f0535611f77fc0edfffe0300c8"}
Nov 03 20:32:04.499138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.499110    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z" event={"ID":"48c9c490-efc4-4157-830b-8c3bb6bee5ce","Type":"ContainerStarted","Data":"922f035d1c23ef68c3fc7943721b99ad8caaceb911d384c6297a84041a5d0d6a"}
Nov 03 20:32:04.499265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.499138    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z" podUID="48c9c490-efc4-4157-830b-8c3bb6bee5ce" containerName="revoke-keys" containerID="cri-o://922f035d1c23ef68c3fc7943721b99ad8caaceb911d384c6297a84041a5d0d6a" gracePeriod=30
Nov 03 20:32:04.499265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.499192    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z" event={"ID":"48c9c490-efc4-4157-830b-8c3bb6bee5ce","Type":"ContainerStarted","Data":"3de0047a4b64eaa09d9f36f635c497fdfe091d5331eb9f80eed4e4eb17b75709"}
Nov 03 20:32:04.634796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.627999    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:04.651312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.651250    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z" podStartSLOduration=2.651225857 podStartE2EDuration="2.651225857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:04.629205963 +0000 UTC m=+3276.784108384" watchObservedRunningTime="2026-11-03 20:32:04.651225857 +0000 UTC m=+3276.806128278"
Nov 03 20:32:04.652297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.652269    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"]
Nov 03 20:32:04.712083 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.712014    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65" podStartSLOduration=2.711998923 podStartE2EDuration="2.711998923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:04.66946513 +0000 UTC m=+3276.824367550" watchObservedRunningTime="2026-11-03 20:32:04.711998923 +0000 UTC m=+3276.866901377"
Nov 03 20:32:04.721517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.721247    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5" podStartSLOduration=3.721228835 podStartE2EDuration="3.721228835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:04.709448333 +0000 UTC m=+3276.864350776" watchObservedRunningTime="2026-11-03 20:32:04.721228835 +0000 UTC m=+3276.876131254"
Nov 03 20:32:04.743201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:04.743073    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt" podStartSLOduration=2.743053864 podStartE2EDuration="2.743053864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:04.739730016 +0000 UTC m=+3276.894632439" watchObservedRunningTime="2026-11-03 20:32:04.743053864 +0000 UTC m=+3276.897956285"
Nov 03 20:32:05.141797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.141763    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"
Nov 03 20:32:05.221520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.221321    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"
Nov 03 20:32:05.237233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.235421    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c9c490-efc4-4157-830b-8c3bb6bee5ce-kube-api-access-qld6c\" (UniqueName: \"kubernetes.io/projected/48c9c490-efc4-4157-830b-8c3bb6bee5ce-kube-api-access-qld6c\") pod \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\" (UID: \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\") "
Nov 03 20:32:05.237233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.235511    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-openshift-ca\") pod \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\" (UID: \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\") "
Nov 03 20:32:05.237233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.235588    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-xks-ca\") pod \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\" (UID: \"48c9c490-efc4-4157-830b-8c3bb6bee5ce\") "
Nov 03 20:32:05.237233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.236179    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-openshift-ca" pod "48c9c490-efc4-4157-830b-8c3bb6bee5ce" (UID: "48c9c490-efc4-4157-830b-8c3bb6bee5ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:05.238758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.238729    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:05.244908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.244876    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c9c490-efc4-4157-830b-8c3bb6bee5ce-kube-api-access-qld6c" pod "48c9c490-efc4-4157-830b-8c3bb6bee5ce" (UID: "48c9c490-efc4-4157-830b-8c3bb6bee5ce"). InnerVolumeSpecName "kube-api-access-qld6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:05.245161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.245141    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-xks-ca" pod "48c9c490-efc4-4157-830b-8c3bb6bee5ce" (UID: "48c9c490-efc4-4157-830b-8c3bb6bee5ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:05.340942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.340564    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ecbc99-1b53-4a25-8944-20f24f9a4044-kube-api-access-2p5n7\" (UniqueName: \"kubernetes.io/projected/f4ecbc99-1b53-4a25-8944-20f24f9a4044-kube-api-access-2p5n7\") pod \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\" (UID: \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\") "
Nov 03 20:32:05.340942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.340643    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-xks-ca\") pod \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\" (UID: \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\") "
Nov 03 20:32:05.342586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.340784    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-openshift-ca\") pod \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\" (UID: \"f4ecbc99-1b53-4a25-8944-20f24f9a4044\") "
Nov 03 20:32:05.343152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.342907    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qld6c\" (UniqueName: \"kubernetes.io/projected/48c9c490-efc4-4157-830b-8c3bb6bee5ce-kube-api-access-qld6c\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:05.343152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.342938    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48c9c490-efc4-4157-830b-8c3bb6bee5ce-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:05.343604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.343560    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-openshift-ca" pod "f4ecbc99-1b53-4a25-8944-20f24f9a4044" (UID: "f4ecbc99-1b53-4a25-8944-20f24f9a4044"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:05.350294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.350253    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ecbc99-1b53-4a25-8944-20f24f9a4044-kube-api-access-2p5n7" pod "f4ecbc99-1b53-4a25-8944-20f24f9a4044" (UID: "f4ecbc99-1b53-4a25-8944-20f24f9a4044"). InnerVolumeSpecName "kube-api-access-2p5n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:05.365506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.365440    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-xks-ca" pod "f4ecbc99-1b53-4a25-8944-20f24f9a4044" (UID: "f4ecbc99-1b53-4a25-8944-20f24f9a4044"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:05.445292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.445244    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p5n7\" (UniqueName: \"kubernetes.io/projected/f4ecbc99-1b53-4a25-8944-20f24f9a4044-kube-api-access-2p5n7\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:05.445292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.445280    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:05.445292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.445290    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ecbc99-1b53-4a25-8944-20f24f9a4044-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:05.512263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.512221    5659 generic.go:356] "Generic (PLEG): container finished" podID="f4ecbc99-1b53-4a25-8944-20f24f9a4044" containerID="8c1ccdc689aadd84c46877402beb5ba85fc222a2dc9e485750614e7d1f98df08" exitCode=22
Nov 03 20:32:05.512441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.512289    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"
Nov 03 20:32:05.512441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.512408    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5" event={"ID":"f4ecbc99-1b53-4a25-8944-20f24f9a4044","Type":"ContainerDied","Data":"8c1ccdc689aadd84c46877402beb5ba85fc222a2dc9e485750614e7d1f98df08"}
Nov 03 20:32:05.512620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.512456    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5" event={"ID":"f4ecbc99-1b53-4a25-8944-20f24f9a4044","Type":"ContainerDied","Data":"a623f9c94094519ce621961f28f4e3fa238c26be8eedc51c89627498ce007158"}
Nov 03 20:32:05.512620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.512530    5659 scope.go:122] "RemoveContainer" containerID="8c1ccdc689aadd84c46877402beb5ba85fc222a2dc9e485750614e7d1f98df08"
Nov 03 20:32:05.514672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.514645    5659 generic.go:356] "Generic (PLEG): container finished" podID="48c9c490-efc4-4157-830b-8c3bb6bee5ce" containerID="922f035d1c23ef68c3fc7943721b99ad8caaceb911d384c6297a84041a5d0d6a" exitCode=22
Nov 03 20:32:05.514798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.514762    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z" event={"ID":"48c9c490-efc4-4157-830b-8c3bb6bee5ce","Type":"ContainerDied","Data":"922f035d1c23ef68c3fc7943721b99ad8caaceb911d384c6297a84041a5d0d6a"}
Nov 03 20:32:05.514798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.514795    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z" event={"ID":"48c9c490-efc4-4157-830b-8c3bb6bee5ce","Type":"ContainerDied","Data":"3de0047a4b64eaa09d9f36f635c497fdfe091d5331eb9f80eed4e4eb17b75709"}
Nov 03 20:32:05.514940 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.514864    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"
Nov 03 20:32:05.528277 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.528252    5659 scope.go:122] "RemoveContainer" containerID="92fc3947aa9412ec8c28b36caaeef30a9a374d42c3b2e377aef46c91f002f55d"
Nov 03 20:32:05.532243 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.531984    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:32:05.532243 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.532036    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f" podUID="b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:32:05.566651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.566620    5659 scope.go:122] "RemoveContainer" containerID="8c1ccdc689aadd84c46877402beb5ba85fc222a2dc9e485750614e7d1f98df08"
Nov 03 20:32:05.567041 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:05.567012    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1ccdc689aadd84c46877402beb5ba85fc222a2dc9e485750614e7d1f98df08\": container with ID starting with 8c1ccdc689aadd84c46877402beb5ba85fc222a2dc9e485750614e7d1f98df08 not found: ID does not exist" containerID="8c1ccdc689aadd84c46877402beb5ba85fc222a2dc9e485750614e7d1f98df08"
Nov 03 20:32:05.567143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.567053    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1ccdc689aadd84c46877402beb5ba85fc222a2dc9e485750614e7d1f98df08"} err="failed to get container status \"8c1ccdc689aadd84c46877402beb5ba85fc222a2dc9e485750614e7d1f98df08\": rpc error: code = NotFound desc = could not find container \"8c1ccdc689aadd84c46877402beb5ba85fc222a2dc9e485750614e7d1f98df08\": container with ID starting with 8c1ccdc689aadd84c46877402beb5ba85fc222a2dc9e485750614e7d1f98df08 not found: ID does not exist"
Nov 03 20:32:05.567143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.567079    5659 scope.go:122] "RemoveContainer" containerID="92fc3947aa9412ec8c28b36caaeef30a9a374d42c3b2e377aef46c91f002f55d"
Nov 03 20:32:05.568535 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:05.568504    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92fc3947aa9412ec8c28b36caaeef30a9a374d42c3b2e377aef46c91f002f55d\": container with ID starting with 92fc3947aa9412ec8c28b36caaeef30a9a374d42c3b2e377aef46c91f002f55d not found: ID does not exist" containerID="92fc3947aa9412ec8c28b36caaeef30a9a374d42c3b2e377aef46c91f002f55d"
Nov 03 20:32:05.568643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.568583    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fc3947aa9412ec8c28b36caaeef30a9a374d42c3b2e377aef46c91f002f55d"} err="failed to get container status \"92fc3947aa9412ec8c28b36caaeef30a9a374d42c3b2e377aef46c91f002f55d\": rpc error: code = NotFound desc = could not find container \"92fc3947aa9412ec8c28b36caaeef30a9a374d42c3b2e377aef46c91f002f55d\": container with ID starting with 92fc3947aa9412ec8c28b36caaeef30a9a374d42c3b2e377aef46c91f002f55d not found: ID does not exist"
Nov 03 20:32:05.568643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.568608    5659 scope.go:122] "RemoveContainer" containerID="922f035d1c23ef68c3fc7943721b99ad8caaceb911d384c6297a84041a5d0d6a"
Nov 03 20:32:05.569211 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.569185    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"]
Nov 03 20:32:05.582463 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.581771    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-z8t7z"]
Nov 03 20:32:05.588369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.588343    5659 scope.go:122] "RemoveContainer" containerID="922f035d1c23ef68c3fc7943721b99ad8caaceb911d384c6297a84041a5d0d6a"
Nov 03 20:32:05.589001 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:05.588980    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922f035d1c23ef68c3fc7943721b99ad8caaceb911d384c6297a84041a5d0d6a\": container with ID starting with 922f035d1c23ef68c3fc7943721b99ad8caaceb911d384c6297a84041a5d0d6a not found: ID does not exist" containerID="922f035d1c23ef68c3fc7943721b99ad8caaceb911d384c6297a84041a5d0d6a"
Nov 03 20:32:05.589066 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.589010    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922f035d1c23ef68c3fc7943721b99ad8caaceb911d384c6297a84041a5d0d6a"} err="failed to get container status \"922f035d1c23ef68c3fc7943721b99ad8caaceb911d384c6297a84041a5d0d6a\": rpc error: code = NotFound desc = could not find container \"922f035d1c23ef68c3fc7943721b99ad8caaceb911d384c6297a84041a5d0d6a\": container with ID starting with 922f035d1c23ef68c3fc7943721b99ad8caaceb911d384c6297a84041a5d0d6a not found: ID does not exist"
Nov 03 20:32:05.601781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.601722    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"]
Nov 03 20:32:05.639386 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.639336    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-r99x5"]
Nov 03 20:32:05.748800 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.748676    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"
Nov 03 20:32:05.751969 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.751831    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:05.860702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.857269    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fde0783-085d-44fe-a55e-18700eaa6e88-kube-api-access-zbknp\" (UniqueName: \"kubernetes.io/projected/6fde0783-085d-44fe-a55e-18700eaa6e88-kube-api-access-zbknp\") pod \"6fde0783-085d-44fe-a55e-18700eaa6e88\" (UID: \"6fde0783-085d-44fe-a55e-18700eaa6e88\") "
Nov 03 20:32:05.860702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.857342    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-openshift-ca\") pod \"6fde0783-085d-44fe-a55e-18700eaa6e88\" (UID: \"6fde0783-085d-44fe-a55e-18700eaa6e88\") "
Nov 03 20:32:05.860702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.857496    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-xks-ca\") pod \"6fde0783-085d-44fe-a55e-18700eaa6e88\" (UID: \"6fde0783-085d-44fe-a55e-18700eaa6e88\") "
Nov 03 20:32:05.864051 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.864014    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-xks-ca" pod "6fde0783-085d-44fe-a55e-18700eaa6e88" (UID: "6fde0783-085d-44fe-a55e-18700eaa6e88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:05.868380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.868322    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-openshift-ca" pod "6fde0783-085d-44fe-a55e-18700eaa6e88" (UID: "6fde0783-085d-44fe-a55e-18700eaa6e88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:05.874413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.874382    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fde0783-085d-44fe-a55e-18700eaa6e88-kube-api-access-zbknp" pod "6fde0783-085d-44fe-a55e-18700eaa6e88" (UID: "6fde0783-085d-44fe-a55e-18700eaa6e88"). InnerVolumeSpecName "kube-api-access-zbknp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:05.959801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.959717    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbknp\" (UniqueName: \"kubernetes.io/projected/6fde0783-085d-44fe-a55e-18700eaa6e88-kube-api-access-zbknp\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:05.959801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.959757    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:05.959801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.959774    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fde0783-085d-44fe-a55e-18700eaa6e88-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:05.986461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.986414    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c9c490-efc4-4157-830b-8c3bb6bee5ce" path="/var/lib/kubelet/pods/48c9c490-efc4-4157-830b-8c3bb6bee5ce/volumes"
Nov 03 20:32:05.987143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:05.987122    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ecbc99-1b53-4a25-8944-20f24f9a4044" path="/var/lib/kubelet/pods/f4ecbc99-1b53-4a25-8944-20f24f9a4044/volumes"
Nov 03 20:32:06.057659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:06.057614    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:06.105512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:06.101139    5659 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:32:06.105512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:06.101186    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk" podUID="eb414b62-f84b-4da8-9891-b85f75868dd0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:32:06.248884 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:06.248846    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:06.398220 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:06.398183    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:06.537299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:06.537257    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw" event={"ID":"6fde0783-085d-44fe-a55e-18700eaa6e88","Type":"ContainerDied","Data":"57715ad7069648978d86045b31f9ffc0cf478deeb9b42a5ee40f06e0000cf02a"}
Nov 03 20:32:06.537477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:06.537331    5659 scope.go:122] "RemoveContainer" containerID="301180709be8957d8e6c2acfd12bed75d9e9b9f76f16a7ce8a8fb300336675c4"
Nov 03 20:32:06.537477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:06.537474    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"
Nov 03 20:32:06.543888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:06.543856    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:06.566332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:06.566298    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"]
Nov 03 20:32:06.577376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:06.577344    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-s5pjw"]
Nov 03 20:32:06.958797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:06.952997    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:07.550416 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:07.550375    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66b59d468d-j8ndl" event={"ID":"19730c58-ac26-42dc-bd3a-8ec0f6076264","Type":"ContainerStarted","Data":"592b3b2ba7d90c6ec12b9c5604cb25bf0180b0e874d3504e7cfeb569992ab3eb"}
Nov 03 20:32:07.584285 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:07.584216    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-66b59d468d-j8ndl" podStartSLOduration=2.650757961 podStartE2EDuration="5.584196908s" totalImagesPullingTime="2.933438947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:07.576398545 +0000 UTC m=+3279.731300976" watchObservedRunningTime="2026-11-03 20:32:07.584196908 +0000 UTC m=+3279.739099328"
Nov 03 20:32:07.978404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:07.978356    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88" path="/var/lib/kubelet/pods/6fde0783-085d-44fe-a55e-18700eaa6e88/volumes"
Nov 03 20:32:07.980412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:07.980386    5659 scope.go:122] "RemoveContainer" containerID="609da6cab4c5867461872e8c9b30b9ee63f8837e5145853d7f887333604b0075"
Nov 03 20:32:08.081138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:08.081032    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:08.511223 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:08.511181    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47" podUID="f6db9932-e1b0-4a21-9874-4c63d3b40a10" containerName="payload-processing" containerID="cri-o://8d943ed7e272ada8fa824eab8dd5b58e0bc74bfc38a2454995e07b779eb5736e" gracePeriod=25
Nov 03 20:32:08.921286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:08.921257    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:32:09.060387 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.060353    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-socket\") pod \"eb414b62-f84b-4da8-9891-b85f75868dd0\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") "
Nov 03 20:32:09.060646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.060626    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-podinfo\") pod \"eb414b62-f84b-4da8-9891-b85f75868dd0\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") "
Nov 03 20:32:09.060806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.060790    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-certs\") pod \"eb414b62-f84b-4da8-9891-b85f75868dd0\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") "
Nov 03 20:32:09.060994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.060981    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb414b62-f84b-4da8-9891-b85f75868dd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb414b62-f84b-4da8-9891-b85f75868dd0-istiod-ca-cert\") pod \"eb414b62-f84b-4da8-9891-b85f75868dd0\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") "
Nov 03 20:32:09.061199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.061183    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-token\" (UniqueName: \"kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-token\") pod \"eb414b62-f84b-4da8-9891-b85f75868dd0\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") "
Nov 03 20:32:09.061370 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.061347    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-data\") pod \"eb414b62-f84b-4da8-9891-b85f75868dd0\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") "
Nov 03 20:32:09.061667 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.061645    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-credential-socket\") pod \"eb414b62-f84b-4da8-9891-b85f75868dd0\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") "
Nov 03 20:32:09.061919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.061900    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-kube-api-access-jlsjs\" (UniqueName: \"kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-kube-api-access-jlsjs\") pod \"eb414b62-f84b-4da8-9891-b85f75868dd0\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") "
Nov 03 20:32:09.062126 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.062109    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-envoy\") pod \"eb414b62-f84b-4da8-9891-b85f75868dd0\" (UID: \"eb414b62-f84b-4da8-9891-b85f75868dd0\") "
Nov 03 20:32:09.064221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.064197    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-socket" pod "eb414b62-f84b-4da8-9891-b85f75868dd0" (UID: "eb414b62-f84b-4da8-9891-b85f75868dd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.064992 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.064964    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-credential-socket" pod "eb414b62-f84b-4da8-9891-b85f75868dd0" (UID: "eb414b62-f84b-4da8-9891-b85f75868dd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.065264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.065243    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb414b62-f84b-4da8-9891-b85f75868dd0-istiod-ca-cert" pod "eb414b62-f84b-4da8-9891-b85f75868dd0" (UID: "eb414b62-f84b-4da8-9891-b85f75868dd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:09.065865 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.065775    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-certs" pod "eb414b62-f84b-4da8-9891-b85f75868dd0" (UID: "eb414b62-f84b-4da8-9891-b85f75868dd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.067178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.067157    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-data" pod "eb414b62-f84b-4da8-9891-b85f75868dd0" (UID: "eb414b62-f84b-4da8-9891-b85f75868dd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.078759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.078716    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-envoy" pod "eb414b62-f84b-4da8-9891-b85f75868dd0" (UID: "eb414b62-f84b-4da8-9891-b85f75868dd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.080312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.080277    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-kube-api-access-jlsjs" pod "eb414b62-f84b-4da8-9891-b85f75868dd0" (UID: "eb414b62-f84b-4da8-9891-b85f75868dd0"). InnerVolumeSpecName "kube-api-access-jlsjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:09.108867 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.108821    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-podinfo" pod "eb414b62-f84b-4da8-9891-b85f75868dd0" (UID: "eb414b62-f84b-4da8-9891-b85f75868dd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:09.118028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.117980    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-token" pod "eb414b62-f84b-4da8-9891-b85f75868dd0" (UID: "eb414b62-f84b-4da8-9891-b85f75868dd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:09.213081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.182363    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb414b62-f84b-4da8-9891-b85f75868dd0-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.213081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.182409    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.213081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.182425    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.213081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.182446    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.213081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.182461    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlsjs\" (UniqueName: \"kubernetes.io/projected/eb414b62-f84b-4da8-9891-b85f75868dd0-kube-api-access-jlsjs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.213081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.182477    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.213081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.182507    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.213081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.182521    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb414b62-f84b-4da8-9891-b85f75868dd0-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.213081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.182535    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb414b62-f84b-4da8-9891-b85f75868dd0-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.282397 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.279995    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:09.294304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.286234    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:32:09.294304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.291690    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"
Nov 03 20:32:09.297321 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.297102    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:32:09.301694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.301663    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:32:09.385014 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.384966    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-data\") pod \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") "
Nov 03 20:32:09.385284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385030    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-token\" (UniqueName: \"kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-token\") pod \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") "
Nov 03 20:32:09.385284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385088    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6db9932-e1b0-4a21-9874-4c63d3b40a10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6db9932-e1b0-4a21-9874-4c63d3b40a10-plugins-config-volume\") pod \"f6db9932-e1b0-4a21-9874-4c63d3b40a10\" (UID: \"f6db9932-e1b0-4a21-9874-4c63d3b40a10\") "
Nov 03 20:32:09.385284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385168    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-kube-api-access-vrgxs\" (UniqueName: \"kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-kube-api-access-vrgxs\") pod \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") "
Nov 03 20:32:09.385284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385228    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-kube-api-access-zq4g9\" (UniqueName: \"kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-kube-api-access-zq4g9\") pod \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") "
Nov 03 20:32:09.385540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385331    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-token\" (UniqueName: \"kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-token\") pod \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") "
Nov 03 20:32:09.385540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385396    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istiod-ca-cert\") pod \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") "
Nov 03 20:32:09.385540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385425    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-envoy\") pod \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") "
Nov 03 20:32:09.385540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385467    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-kube-api-access-vd9vc\" (UniqueName: \"kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-kube-api-access-vd9vc\") pod \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") "
Nov 03 20:32:09.385540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385530    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-credential-socket\") pod \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") "
Nov 03 20:32:09.385711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385567    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-data\") pod \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") "
Nov 03 20:32:09.385711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385652    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-data\") pod \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") "
Nov 03 20:32:09.385711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385689    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-socket\") pod \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") "
Nov 03 20:32:09.385807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385735    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-certs\") pod \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") "
Nov 03 20:32:09.385843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385815    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-envoy\") pod \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") "
Nov 03 20:32:09.385876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385851    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-certs\") pod \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") "
Nov 03 20:32:09.385925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385907    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-podinfo\") pod \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") "
Nov 03 20:32:09.385962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385951    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-podinfo\") pod \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") "
Nov 03 20:32:09.386007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.385992    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-socket\") pod \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") "
Nov 03 20:32:09.386048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.386035    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istiod-ca-cert\") pod \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") "
Nov 03 20:32:09.386113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.386099    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-socket\") pod \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") "
Nov 03 20:32:09.386206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.386192    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-envoy\") pod \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") "
Nov 03 20:32:09.386319 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.386255    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-token\" (UniqueName: \"kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-token\") pod \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") "
Nov 03 20:32:09.386364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.386319    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istiod-ca-cert\") pod \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") "
Nov 03 20:32:09.386397 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.386361    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-credential-socket\") pod \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\" (UID: \"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e\") "
Nov 03 20:32:09.386430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.386395    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-certs\") pod \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\" (UID: \"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1\") "
Nov 03 20:32:09.386516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.386468    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-podinfo\") pod \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") "
Nov 03 20:32:09.386574 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.386523    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-credential-socket\") pod \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\" (UID: \"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e\") "
Nov 03 20:32:09.386629 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.386578    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6db9932-e1b0-4a21-9874-4c63d3b40a10-kube-api-access-gdtwd\" (UniqueName: \"kubernetes.io/projected/f6db9932-e1b0-4a21-9874-4c63d3b40a10-kube-api-access-gdtwd\") pod \"f6db9932-e1b0-4a21-9874-4c63d3b40a10\" (UID: \"f6db9932-e1b0-4a21-9874-4c63d3b40a10\") "
Nov 03 20:32:09.386936 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.386908    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-certs" pod "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" (UID: "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.388158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.387539    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istiod-ca-cert" pod "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" (UID: "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:09.388158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.387588    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.388158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.387879    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-data" pod "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" (UID: "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.390078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.389784    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6db9932-e1b0-4a21-9874-4c63d3b40a10-plugins-config-volume" pod "f6db9932-e1b0-4a21-9874-4c63d3b40a10" (UID: "f6db9932-e1b0-4a21-9874-4c63d3b40a10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:09.390078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.389819    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6db9932-e1b0-4a21-9874-4c63d3b40a10-kube-api-access-gdtwd" pod "f6db9932-e1b0-4a21-9874-4c63d3b40a10" (UID: "f6db9932-e1b0-4a21-9874-4c63d3b40a10"). InnerVolumeSpecName "kube-api-access-gdtwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:09.390603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.390310    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-data" pod "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" (UID: "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.390706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.390613    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-token" pod "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" (UID: "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:09.394893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.391821    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-socket" pod "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" (UID: "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.394893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.392204    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-credential-socket" pod "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" (UID: "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.394893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.394003    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-kube-api-access-vrgxs" pod "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" (UID: "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e"). InnerVolumeSpecName "kube-api-access-vrgxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:09.406971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.395601    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-kube-api-access-zq4g9" pod "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" (UID: "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1"). InnerVolumeSpecName "kube-api-access-zq4g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:09.406971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.399633    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-envoy" pod "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" (UID: "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.406971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.402937    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istiod-ca-cert" pod "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" (UID: "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:09.406971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.403202    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-socket" pod "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" (UID: "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.406971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.403408    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-certs" pod "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" (UID: "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.406971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.403709    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istiod-ca-cert" pod "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" (UID: "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:09.406971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.403940    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-credential-socket" pod "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" (UID: "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.406971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.404107    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-socket" pod "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" (UID: "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.406971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.405519    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-data" pod "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" (UID: "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.406971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.406833    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-credential-socket" pod "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" (UID: "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.417916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.410032    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-certs" pod "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" (UID: "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.417916 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.417797    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-token" pod "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" (UID: "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:09.419576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.419533    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-envoy" pod "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" (UID: "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.427167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.427103    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-envoy" pod "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" (UID: "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.440063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.439995    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-podinfo" pod "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" (UID: "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:09.440511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.440448    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-token" pod "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" (UID: "3c35f36b-9f8c-4dcc-84aa-f3c17b28252e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:09.440847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.440808    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-podinfo" pod "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" (UID: "b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:09.441193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.441150    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-kube-api-access-vd9vc" pod "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" (UID: "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e"). InnerVolumeSpecName "kube-api-access-vd9vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:09.444157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.444116    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-podinfo" pod "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" (UID: "0e0fa848-0fe6-41d8-b068-8bd42f81bb6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:09.490431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490387    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490434    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490449    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490465    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490498    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490513    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490526    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490542    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490556    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490569    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490583    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490598    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdtwd\" (UniqueName: \"kubernetes.io/projected/f6db9932-e1b0-4a21-9874-4c63d3b40a10-kube-api-access-gdtwd\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490613    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490628    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490644    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6db9932-e1b0-4a21-9874-4c63d3b40a10-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490658    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrgxs\" (UniqueName: \"kubernetes.io/projected/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-kube-api-access-vrgxs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490674    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq4g9\" (UniqueName: \"kubernetes.io/projected/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-kube-api-access-zq4g9\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490687    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490701    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490714    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490728    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd9vc\" (UniqueName: \"kubernetes.io/projected/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-kube-api-access-vd9vc\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490742    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.490751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490757    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.491899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490771    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.491899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490799    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.491899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490812    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.491899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490826    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.491899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.490841    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.491899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.491464    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65" podUID="78765763-73ba-4537-aa8a-47b8159a58c6" containerName="payload-processing" containerID="cri-o://de87f3fa0926cd55169892f5dca7b2a6f75772887ada5b42ce946e5aa2529d3c" gracePeriod=25
Nov 03 20:32:09.589154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.589106    5659 generic.go:356] "Generic (PLEG): container finished" podID="b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" containerID="5976e2e2d71acbbf7af0fd68d94b85ed4aeb64c1a30ebc81c14b5ee275d8a052" exitCode=0
Nov 03 20:32:09.589345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.589173    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"
Nov 03 20:32:09.589345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.589199    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f" event={"ID":"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1","Type":"ContainerDied","Data":"5976e2e2d71acbbf7af0fd68d94b85ed4aeb64c1a30ebc81c14b5ee275d8a052"}
Nov 03 20:32:09.589345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.589258    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f" event={"ID":"b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1","Type":"ContainerDied","Data":"60dff2036018da1d7385a424ec683e78ee8b204a06a54a165a648a0253c05928"}
Nov 03 20:32:09.589345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.589282    5659 scope.go:122] "RemoveContainer" containerID="5976e2e2d71acbbf7af0fd68d94b85ed4aeb64c1a30ebc81c14b5ee275d8a052"
Nov 03 20:32:09.591134 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.591098    5659 generic.go:356] "Generic (PLEG): container finished" podID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" containerID="411a7f77513285e0bd3d10efb914958e55485b6921f9300096f36d8817aff58c" exitCode=0
Nov 03 20:32:09.591254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.591135    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" event={"ID":"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e","Type":"ContainerDied","Data":"411a7f77513285e0bd3d10efb914958e55485b6921f9300096f36d8817aff58c"}
Nov 03 20:32:09.591254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.591191    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b" event={"ID":"3c35f36b-9f8c-4dcc-84aa-f3c17b28252e","Type":"ContainerDied","Data":"03ae979361760fc54635284110afd60c1f2ed6ded9e24ef6b1f9aede621a326d"}
Nov 03 20:32:09.591254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.591147    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"
Nov 03 20:32:09.593580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.593556    5659 generic.go:356] "Generic (PLEG): container finished" podID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerID="df64b065a07a619c7cf3705172be7c7543e8d9685ba16888bcb9054792945514" exitCode=0
Nov 03 20:32:09.593683 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.593599    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"
Nov 03 20:32:09.593746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.593679    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" event={"ID":"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e","Type":"ContainerDied","Data":"df64b065a07a619c7cf3705172be7c7543e8d9685ba16888bcb9054792945514"}
Nov 03 20:32:09.593746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.593731    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz" event={"ID":"0e0fa848-0fe6-41d8-b068-8bd42f81bb6e","Type":"ContainerDied","Data":"07b6cdde9cce4df8b8524aadf057c6db70876adee741f339fd9264cb99bbea06"}
Nov 03 20:32:09.606845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.602301    5659 generic.go:356] "Generic (PLEG): container finished" podID="eb414b62-f84b-4da8-9891-b85f75868dd0" containerID="2af7ed40bd868991cc818d3c07409189994e01ea4556d31ecc0be76fc17e32a4" exitCode=0
Nov 03 20:32:09.606845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.602474    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk" event={"ID":"eb414b62-f84b-4da8-9891-b85f75868dd0","Type":"ContainerDied","Data":"2af7ed40bd868991cc818d3c07409189994e01ea4556d31ecc0be76fc17e32a4"}
Nov 03 20:32:09.606845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.602583    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk" event={"ID":"eb414b62-f84b-4da8-9891-b85f75868dd0","Type":"ContainerDied","Data":"9250d8aa2d30caa19136e9e7d7efc70358e76c77a16296ab5674f977f7ab042f"}
Nov 03 20:32:09.606845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.602673    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"
Nov 03 20:32:09.612521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.610044    5659 generic.go:356] "Generic (PLEG): container finished" podID="bc8b2605-2520-455c-aa28-6d81b5601ab8" containerID="5b150933485f88c93209842e1d211cde05aee383bd2e107020685169fb715b35" exitCode=7
Nov 03 20:32:09.612521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.610172    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c" event={"ID":"bc8b2605-2520-455c-aa28-6d81b5601ab8","Type":"ContainerDied","Data":"5b150933485f88c93209842e1d211cde05aee383bd2e107020685169fb715b35"}
Nov 03 20:32:09.612521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.610692    5659 scope.go:122] "RemoveContainer" containerID="5b150933485f88c93209842e1d211cde05aee383bd2e107020685169fb715b35"
Nov 03 20:32:09.612521 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:09.611122    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-thg6c_odh-ai-gateway-infra(bc8b2605-2520-455c-aa28-6d81b5601ab8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c" podUID="bc8b2605-2520-455c-aa28-6d81b5601ab8"
Nov 03 20:32:09.617907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.614670    5659 generic.go:356] "Generic (PLEG): container finished" podID="f6db9932-e1b0-4a21-9874-4c63d3b40a10" containerID="8d943ed7e272ada8fa824eab8dd5b58e0bc74bfc38a2454995e07b779eb5736e" exitCode=0
Nov 03 20:32:09.617907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.614721    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47" event={"ID":"f6db9932-e1b0-4a21-9874-4c63d3b40a10","Type":"ContainerDied","Data":"8d943ed7e272ada8fa824eab8dd5b58e0bc74bfc38a2454995e07b779eb5736e"}
Nov 03 20:32:09.617907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.614761    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47" event={"ID":"f6db9932-e1b0-4a21-9874-4c63d3b40a10","Type":"ContainerDied","Data":"cd43bab1bfe92cb1f74df34fccddb6f43b603423af29b3e29af5b6ba94e520d9"}
Nov 03 20:32:09.617907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.614831    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"
Nov 03 20:32:09.646771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.645768    5659 scope.go:122] "RemoveContainer" containerID="5976e2e2d71acbbf7af0fd68d94b85ed4aeb64c1a30ebc81c14b5ee275d8a052"
Nov 03 20:32:09.647340 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:09.647313    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5976e2e2d71acbbf7af0fd68d94b85ed4aeb64c1a30ebc81c14b5ee275d8a052\": container with ID starting with 5976e2e2d71acbbf7af0fd68d94b85ed4aeb64c1a30ebc81c14b5ee275d8a052 not found: ID does not exist" containerID="5976e2e2d71acbbf7af0fd68d94b85ed4aeb64c1a30ebc81c14b5ee275d8a052"
Nov 03 20:32:09.697987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.688581    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5976e2e2d71acbbf7af0fd68d94b85ed4aeb64c1a30ebc81c14b5ee275d8a052"} err="failed to get container status \"5976e2e2d71acbbf7af0fd68d94b85ed4aeb64c1a30ebc81c14b5ee275d8a052\": rpc error: code = NotFound desc = could not find container \"5976e2e2d71acbbf7af0fd68d94b85ed4aeb64c1a30ebc81c14b5ee275d8a052\": container with ID starting with 5976e2e2d71acbbf7af0fd68d94b85ed4aeb64c1a30ebc81c14b5ee275d8a052 not found: ID does not exist"
Nov 03 20:32:09.697987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.688635    5659 scope.go:122] "RemoveContainer" containerID="411a7f77513285e0bd3d10efb914958e55485b6921f9300096f36d8817aff58c"
Nov 03 20:32:09.714988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.714947    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"]
Nov 03 20:32:09.728567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.726338    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-b7d9f"]
Nov 03 20:32:09.733960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.733908    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"]
Nov 03 20:32:09.746033 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.745993    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-mrnwk"]
Nov 03 20:32:09.766149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.766112    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"]
Nov 03 20:32:09.796312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.796275    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-74dfc6-587456cfdb-csd47"]
Nov 03 20:32:09.851948 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.851861    5659 scope.go:122] "RemoveContainer" containerID="411a7f77513285e0bd3d10efb914958e55485b6921f9300096f36d8817aff58c"
Nov 03 20:32:09.853800 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:09.853749    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411a7f77513285e0bd3d10efb914958e55485b6921f9300096f36d8817aff58c\": container with ID starting with 411a7f77513285e0bd3d10efb914958e55485b6921f9300096f36d8817aff58c not found: ID does not exist" containerID="411a7f77513285e0bd3d10efb914958e55485b6921f9300096f36d8817aff58c"
Nov 03 20:32:09.853988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.853813    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411a7f77513285e0bd3d10efb914958e55485b6921f9300096f36d8817aff58c"} err="failed to get container status \"411a7f77513285e0bd3d10efb914958e55485b6921f9300096f36d8817aff58c\": rpc error: code = NotFound desc = could not find container \"411a7f77513285e0bd3d10efb914958e55485b6921f9300096f36d8817aff58c\": container with ID starting with 411a7f77513285e0bd3d10efb914958e55485b6921f9300096f36d8817aff58c not found: ID does not exist"
Nov 03 20:32:09.853988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.853860    5659 scope.go:122] "RemoveContainer" containerID="df64b065a07a619c7cf3705172be7c7543e8d9685ba16888bcb9054792945514"
Nov 03 20:32:09.928167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.928131    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"]
Nov 03 20:32:09.939457 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.939407    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qsh2b"]
Nov 03 20:32:09.960057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.960026    5659 scope.go:122] "RemoveContainer" containerID="df64b065a07a619c7cf3705172be7c7543e8d9685ba16888bcb9054792945514"
Nov 03 20:32:09.960730 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:09.960702    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df64b065a07a619c7cf3705172be7c7543e8d9685ba16888bcb9054792945514\": container with ID starting with df64b065a07a619c7cf3705172be7c7543e8d9685ba16888bcb9054792945514 not found: ID does not exist" containerID="df64b065a07a619c7cf3705172be7c7543e8d9685ba16888bcb9054792945514"
Nov 03 20:32:09.960942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.960909    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df64b065a07a619c7cf3705172be7c7543e8d9685ba16888bcb9054792945514"} err="failed to get container status \"df64b065a07a619c7cf3705172be7c7543e8d9685ba16888bcb9054792945514\": rpc error: code = NotFound desc = could not find container \"df64b065a07a619c7cf3705172be7c7543e8d9685ba16888bcb9054792945514\": container with ID starting with df64b065a07a619c7cf3705172be7c7543e8d9685ba16888bcb9054792945514 not found: ID does not exist"
Nov 03 20:32:09.961046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.961033    5659 scope.go:122] "RemoveContainer" containerID="2af7ed40bd868991cc818d3c07409189994e01ea4556d31ecc0be76fc17e32a4"
Nov 03 20:32:09.993858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.991929    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" path="/var/lib/kubelet/pods/3c35f36b-9f8c-4dcc-84aa-f3c17b28252e/volumes"
Nov 03 20:32:09.993858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.992788    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" path="/var/lib/kubelet/pods/b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1/volumes"
Nov 03 20:32:09.993858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.993581    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb414b62-f84b-4da8-9891-b85f75868dd0" path="/var/lib/kubelet/pods/eb414b62-f84b-4da8-9891-b85f75868dd0/volumes"
Nov 03 20:32:09.995023 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.994632    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6db9932-e1b0-4a21-9874-4c63d3b40a10" path="/var/lib/kubelet/pods/f6db9932-e1b0-4a21-9874-4c63d3b40a10/volumes"
Nov 03 20:32:09.995250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.995217    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"]
Nov 03 20:32:09.995410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:09.995397    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-dh8xz"]
Nov 03 20:32:10.042000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.041960    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"
Nov 03 20:32:10.070800 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.068728    5659 scope.go:122] "RemoveContainer" containerID="2af7ed40bd868991cc818d3c07409189994e01ea4556d31ecc0be76fc17e32a4"
Nov 03 20:32:10.071501 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:10.071443    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af7ed40bd868991cc818d3c07409189994e01ea4556d31ecc0be76fc17e32a4\": container with ID starting with 2af7ed40bd868991cc818d3c07409189994e01ea4556d31ecc0be76fc17e32a4 not found: ID does not exist" containerID="2af7ed40bd868991cc818d3c07409189994e01ea4556d31ecc0be76fc17e32a4"
Nov 03 20:32:10.071858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.071738    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af7ed40bd868991cc818d3c07409189994e01ea4556d31ecc0be76fc17e32a4"} err="failed to get container status \"2af7ed40bd868991cc818d3c07409189994e01ea4556d31ecc0be76fc17e32a4\": rpc error: code = NotFound desc = could not find container \"2af7ed40bd868991cc818d3c07409189994e01ea4556d31ecc0be76fc17e32a4\": container with ID starting with 2af7ed40bd868991cc818d3c07409189994e01ea4556d31ecc0be76fc17e32a4 not found: ID does not exist"
Nov 03 20:32:10.071858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.071769    5659 scope.go:122] "RemoveContainer" containerID="609da6cab4c5867461872e8c9b30b9ee63f8837e5145853d7f887333604b0075"
Nov 03 20:32:10.113225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.105871    5659 scope.go:122] "RemoveContainer" containerID="8d943ed7e272ada8fa824eab8dd5b58e0bc74bfc38a2454995e07b779eb5736e"
Nov 03 20:32:10.113225 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.109699    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:10.128507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.126673    5659 scope.go:122] "RemoveContainer" containerID="8d943ed7e272ada8fa824eab8dd5b58e0bc74bfc38a2454995e07b779eb5736e"
Nov 03 20:32:10.128507 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:10.127310    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d943ed7e272ada8fa824eab8dd5b58e0bc74bfc38a2454995e07b779eb5736e\": container with ID starting with 8d943ed7e272ada8fa824eab8dd5b58e0bc74bfc38a2454995e07b779eb5736e not found: ID does not exist" containerID="8d943ed7e272ada8fa824eab8dd5b58e0bc74bfc38a2454995e07b779eb5736e"
Nov 03 20:32:10.128507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.127354    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d943ed7e272ada8fa824eab8dd5b58e0bc74bfc38a2454995e07b779eb5736e"} err="failed to get container status \"8d943ed7e272ada8fa824eab8dd5b58e0bc74bfc38a2454995e07b779eb5736e\": rpc error: code = NotFound desc = could not find container \"8d943ed7e272ada8fa824eab8dd5b58e0bc74bfc38a2454995e07b779eb5736e\": container with ID starting with 8d943ed7e272ada8fa824eab8dd5b58e0bc74bfc38a2454995e07b779eb5736e not found: ID does not exist"
Nov 03 20:32:10.131659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.131370    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78765763-73ba-4537-aa8a-47b8159a58c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78765763-73ba-4537-aa8a-47b8159a58c6-plugins-config-volume\") pod \"78765763-73ba-4537-aa8a-47b8159a58c6\" (UID: \"78765763-73ba-4537-aa8a-47b8159a58c6\") "
Nov 03 20:32:10.131789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.131701    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78765763-73ba-4537-aa8a-47b8159a58c6-kube-api-access-4jlgl\" (UniqueName: \"kubernetes.io/projected/78765763-73ba-4537-aa8a-47b8159a58c6-kube-api-access-4jlgl\") pod \"78765763-73ba-4537-aa8a-47b8159a58c6\" (UID: \"78765763-73ba-4537-aa8a-47b8159a58c6\") "
Nov 03 20:32:10.133302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.132031    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78765763-73ba-4537-aa8a-47b8159a58c6-plugins-config-volume" pod "78765763-73ba-4537-aa8a-47b8159a58c6" (UID: "78765763-73ba-4537-aa8a-47b8159a58c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:10.133302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.132679    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78765763-73ba-4537-aa8a-47b8159a58c6-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:10.161856 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.161802    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78765763-73ba-4537-aa8a-47b8159a58c6-kube-api-access-4jlgl" pod "78765763-73ba-4537-aa8a-47b8159a58c6" (UID: "78765763-73ba-4537-aa8a-47b8159a58c6"). InnerVolumeSpecName "kube-api-access-4jlgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.166687    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"]
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.167968    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.167991    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168014    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168023    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168038    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f4ecbc99-1b53-4a25-8944-20f24f9a4044" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168045    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ecbc99-1b53-4a25-8944-20f24f9a4044" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168074    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f4ecbc99-1b53-4a25-8944-20f24f9a4044" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168082    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ecbc99-1b53-4a25-8944-20f24f9a4044" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168095    5659 cpu_manager.go:414] "containerMap: removing container" podUID="48c9c490-efc4-4157-830b-8c3bb6bee5ce" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168103    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c9c490-efc4-4157-830b-8c3bb6bee5ce" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168128    5659 cpu_manager.go:414] "containerMap: removing container" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168138    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168153    5659 cpu_manager.go:414] "containerMap: removing container" podUID="78765763-73ba-4537-aa8a-47b8159a58c6" containerName="payload-processing"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168163    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78765763-73ba-4537-aa8a-47b8159a58c6" containerName="payload-processing"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168178    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168186    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168202    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f6db9932-e1b0-4a21-9874-4c63d3b40a10" containerName="payload-processing"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168209    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6db9932-e1b0-4a21-9874-4c63d3b40a10" containerName="payload-processing"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168234    5659 cpu_manager.go:414] "containerMap: removing container" podUID="21a70e8c-6980-4dcb-aafd-0cf5feecd588" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168241    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a70e8c-6980-4dcb-aafd-0cf5feecd588" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168253    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168260    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168303    5659 cpu_manager.go:414] "containerMap: removing container" podUID="eb414b62-f84b-4da8-9891-b85f75868dd0" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168310    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb414b62-f84b-4da8-9891-b85f75868dd0" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168518    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a70e8c-6980-4dcb-aafd-0cf5feecd588" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168537    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c35f36b-9f8c-4dcc-84aa-f3c17b28252e" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168559    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168567    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb414b62-f84b-4da8-9891-b85f75868dd0" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168578    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168588    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168597    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ecbc99-1b53-4a25-8944-20f24f9a4044" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168607    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ecbc99-1b53-4a25-8944-20f24f9a4044" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168616    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c9c490-efc4-4157-830b-8c3bb6bee5ce" containerName="revoke-keys"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168627    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168635    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="78765763-73ba-4537-aa8a-47b8159a58c6" containerName="payload-processing"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168645    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76abb9d-37a1-4f23-bd0e-ffd468c5dbe1" containerName="istio-proxy"
Nov 03 20:32:10.172176 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.168653    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6db9932-e1b0-4a21-9874-4c63d3b40a10" containerName="payload-processing"
Nov 03 20:32:10.190096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.186066    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.198832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.196939    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-m2pgc"]
Nov 03 20:32:10.198832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.198196    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88" containerName="revoke-keys"
Nov 03 20:32:10.198832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.198219    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fde0783-085d-44fe-a55e-18700eaa6e88" containerName="revoke-keys"
Nov 03 20:32:10.208082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.207062    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-m2pgc"
Nov 03 20:32:10.209429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.209408    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jhb7v"]
Nov 03 20:32:10.217545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.217514    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jhb7v"
Nov 03 20:32:10.219851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.219740    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 20:32:10.220069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.219818    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-dkxw8\""
Nov 03 20:32:10.224364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.224198    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 20:32:10.227067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.227002    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:32:10.227862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.227329    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 20:32:10.235945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.234903    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-socket\") pod \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") "
Nov 03 20:32:10.235945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.235064    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-certs\") pod \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") "
Nov 03 20:32:10.235945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.235115    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-data\") pod \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") "
Nov 03 20:32:10.235945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.235185    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-kube-api-access-8wghr\" (UniqueName: \"kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-kube-api-access-8wghr\") pod \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") "
Nov 03 20:32:10.235945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.235225    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istiod-ca-cert\") pod \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") "
Nov 03 20:32:10.236256 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.236006    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-certs" pod "21a70e8c-6980-4dcb-aafd-0cf5feecd588" (UID: "21a70e8c-6980-4dcb-aafd-0cf5feecd588"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:10.238325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.236575    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-socket" pod "21a70e8c-6980-4dcb-aafd-0cf5feecd588" (UID: "21a70e8c-6980-4dcb-aafd-0cf5feecd588"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:10.238325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.236643    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-data" pod "21a70e8c-6980-4dcb-aafd-0cf5feecd588" (UID: "21a70e8c-6980-4dcb-aafd-0cf5feecd588"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:10.238325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.236790    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istiod-ca-cert" pod "21a70e8c-6980-4dcb-aafd-0cf5feecd588" (UID: "21a70e8c-6980-4dcb-aafd-0cf5feecd588"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:10.238772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.238748    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-envoy\") pod \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") "
Nov 03 20:32:10.238923 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.238909    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-podinfo\") pod \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") "
Nov 03 20:32:10.239036 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.239024    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-token\" (UniqueName: \"kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-token\") pod \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") "
Nov 03 20:32:10.239187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.239174    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-credential-socket\") pod \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\" (UID: \"21a70e8c-6980-4dcb-aafd-0cf5feecd588\") "
Nov 03 20:32:10.240287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.240203    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-kube-api-access-8wghr" pod "21a70e8c-6980-4dcb-aafd-0cf5feecd588" (UID: "21a70e8c-6980-4dcb-aafd-0cf5feecd588"). InnerVolumeSpecName "kube-api-access-8wghr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:10.240287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.240231    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jlgl\" (UniqueName: \"kubernetes.io/projected/78765763-73ba-4537-aa8a-47b8159a58c6-kube-api-access-4jlgl\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:10.240287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.240253    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:10.240287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.240269    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:10.240287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.240283    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:10.240752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.240296    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:10.240752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.240512    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-credential-socket" pod "21a70e8c-6980-4dcb-aafd-0cf5feecd588" (UID: "21a70e8c-6980-4dcb-aafd-0cf5feecd588"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:10.244050 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.244022    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-token" pod "21a70e8c-6980-4dcb-aafd-0cf5feecd588" (UID: "21a70e8c-6980-4dcb-aafd-0cf5feecd588"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:10.245475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.245425    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-envoy" pod "21a70e8c-6980-4dcb-aafd-0cf5feecd588" (UID: "21a70e8c-6980-4dcb-aafd-0cf5feecd588"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:10.273691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.273644    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-podinfo" pod "21a70e8c-6980-4dcb-aafd-0cf5feecd588" (UID: "21a70e8c-6980-4dcb-aafd-0cf5feecd588"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:10.279502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.279455    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-f7b87d695-dwgqt"]
Nov 03 20:32:10.302730 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.302702    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-4ctkr"]
Nov 03 20:32:10.303073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.303054    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.315053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.315028    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-m2pgc"]
Nov 03 20:32:10.315614 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.315595    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jhb7v"]
Nov 03 20:32:10.315837 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.315812    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-4ctkr"]
Nov 03 20:32:10.315966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.315532    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4ctkr"
Nov 03 20:32:10.316861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.316816    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 20:32:10.317179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.317162    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 20:32:10.317420 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.317404    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xr7jf\""
Nov 03 20:32:10.317704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.317644    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 20:32:10.318103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.318011    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xdpbk\""
Nov 03 20:32:10.322959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.322930    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 20:32:10.324273 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.324255    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"]
Nov 03 20:32:10.334369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.334296    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f7b87d695-dwgqt"]
Nov 03 20:32:10.341584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.341552    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.341724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.341669    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76f8240-ab05-410c-8a60-1d7b8cdde912-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.341724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.341709    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/114f0567-5268-473e-aec5-50869958548c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m2pgc\" (UID: \"114f0567-5268-473e-aec5-50869958548c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m2pgc"
Nov 03 20:32:10.341846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.341784    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rf9q\" (UniqueName: \"kubernetes.io/projected/ece7bfca-bf0e-4422-ad06-1df2ca9a08bf-kube-api-access-7rf9q\") pod \"volume-data-source-validator-58c9f5f46d-jhb7v\" (UID: \"ece7bfca-bf0e-4422-ad06-1df2ca9a08bf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jhb7v"
Nov 03 20:32:10.341846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.341821    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.342011 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.341988    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvzpv\" (UniqueName: \"kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-kube-api-access-wvzpv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.342090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.342074    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.342145 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.342123    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/114f0567-5268-473e-aec5-50869958548c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-m2pgc\" (UID: \"114f0567-5268-473e-aec5-50869958548c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m2pgc"
Nov 03 20:32:10.342202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.342181    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.342265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.342210    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.342472 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.342449    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wghr\" (UniqueName: \"kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-kube-api-access-8wghr\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:10.342596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.342496    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:10.342596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.342514    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:10.342596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.342527    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21a70e8c-6980-4dcb-aafd-0cf5feecd588-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:10.342596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.342542    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21a70e8c-6980-4dcb-aafd-0cf5feecd588-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:10.416778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.416745    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-lsdvf\""
Nov 03 20:32:10.444258 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.444222    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-default-certificate\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.444435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.444294    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-metrics-certs\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.444575 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.444431    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.444575 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.444522    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-service-ca-bundle\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.444690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.444607    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76f8240-ab05-410c-8a60-1d7b8cdde912-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.444690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.444646    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/114f0567-5268-473e-aec5-50869958548c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m2pgc\" (UID: \"114f0567-5268-473e-aec5-50869958548c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m2pgc"
Nov 03 20:32:10.444768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.444752    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rf9q\" (UniqueName: \"kubernetes.io/projected/ece7bfca-bf0e-4422-ad06-1df2ca9a08bf-kube-api-access-7rf9q\") pod \"volume-data-source-validator-58c9f5f46d-jhb7v\" (UID: \"ece7bfca-bf0e-4422-ad06-1df2ca9a08bf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jhb7v"
Nov 03 20:32:10.444808 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.444788    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.444891 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.444871    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzpv\" (UniqueName: \"kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-kube-api-access-wvzpv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.445735 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.445710    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-stats-auth\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.445965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.445944    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.446238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.446212    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.446238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.446220    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/114f0567-5268-473e-aec5-50869958548c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-m2pgc\" (UID: \"114f0567-5268-473e-aec5-50869958548c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m2pgc"
Nov 03 20:32:10.446560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.446537    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.446642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.446569    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.446642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.446601    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw9p5\" (UniqueName: \"kubernetes.io/projected/dcd7fddd-b34d-4947-80fc-b48f082004c8-kube-api-access-kw9p5\") pod \"network-check-source-6f78d59f99-4ctkr\" (UID: \"dcd7fddd-b34d-4947-80fc-b48f082004c8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4ctkr"
Nov 03 20:32:10.446770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.446644    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m54rc\" (UniqueName: \"kubernetes.io/projected/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-kube-api-access-m54rc\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.447454 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.447429    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/114f0567-5268-473e-aec5-50869958548c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-m2pgc\" (UID: \"114f0567-5268-473e-aec5-50869958548c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m2pgc"
Nov 03 20:32:10.448331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.448286    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.448654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.448570    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/114f0567-5268-473e-aec5-50869958548c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m2pgc\" (UID: \"114f0567-5268-473e-aec5-50869958548c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m2pgc"
Nov 03 20:32:10.448843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.448825    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76f8240-ab05-410c-8a60-1d7b8cdde912-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.449294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.449268    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.449451 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.449435    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.460405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.460340    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvzpv\" (UniqueName: \"kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-kube-api-access-wvzpv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.461613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.461590    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2vf4r\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.463453 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.463426    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rf9q\" (UniqueName: \"kubernetes.io/projected/ece7bfca-bf0e-4422-ad06-1df2ca9a08bf-kube-api-access-7rf9q\") pod \"volume-data-source-validator-58c9f5f46d-jhb7v\" (UID: \"ece7bfca-bf0e-4422-ad06-1df2ca9a08bf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jhb7v"
Nov 03 20:32:10.516116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.516077    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:10.543004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.542228    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-m2pgc"
Nov 03 20:32:10.587363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.548967    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-stats-auth\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.587363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.549133    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw9p5\" (UniqueName: \"kubernetes.io/projected/dcd7fddd-b34d-4947-80fc-b48f082004c8-kube-api-access-kw9p5\") pod \"network-check-source-6f78d59f99-4ctkr\" (UID: \"dcd7fddd-b34d-4947-80fc-b48f082004c8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4ctkr"
Nov 03 20:32:10.587363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.549189    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m54rc\" (UniqueName: \"kubernetes.io/projected/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-kube-api-access-m54rc\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.587363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.549251    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-default-certificate\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.587363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.549304    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-metrics-certs\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.587363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.549412    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-service-ca-bundle\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.587363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.550402    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-service-ca-bundle\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.587363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.559939    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-default-certificate\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.587363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.565883    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:10.611499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.611412    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m54rc\" (UniqueName: \"kubernetes.io/projected/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-kube-api-access-m54rc\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.630312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.612237    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-metrics-certs\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.652366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.652337    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jhb7v"
Nov 03 20:32:10.657569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.655115    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89-stats-auth\") pod \"router-default-f7b87d695-dwgqt\" (UID: \"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89\") " pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.659199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.659132    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw9p5\" (UniqueName: \"kubernetes.io/projected/dcd7fddd-b34d-4947-80fc-b48f082004c8-kube-api-access-kw9p5\") pod \"network-check-source-6f78d59f99-4ctkr\" (UID: \"dcd7fddd-b34d-4947-80fc-b48f082004c8\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4ctkr"
Nov 03 20:32:10.692004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.686370    5659 generic.go:356] "Generic (PLEG): container finished" podID="21a70e8c-6980-4dcb-aafd-0cf5feecd588" containerID="11fb6de88e3166bb3ed3332661c84de96f809feada507d5f5678faae9959baaa" exitCode=0
Nov 03 20:32:10.692004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.686503    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt" event={"ID":"21a70e8c-6980-4dcb-aafd-0cf5feecd588","Type":"ContainerDied","Data":"11fb6de88e3166bb3ed3332661c84de96f809feada507d5f5678faae9959baaa"}
Nov 03 20:32:10.692004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.686556    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt" event={"ID":"21a70e8c-6980-4dcb-aafd-0cf5feecd588","Type":"ContainerDied","Data":"4484e058c08af1933d7c05749283441abdbf143604ec572a3976621e312ddc2d"}
Nov 03 20:32:10.692004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.686591    5659 scope.go:122] "RemoveContainer" containerID="11fb6de88e3166bb3ed3332661c84de96f809feada507d5f5678faae9959baaa"
Nov 03 20:32:10.692004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.686726    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"
Nov 03 20:32:10.698336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.698298    5659 generic.go:356] "Generic (PLEG): container finished" podID="78765763-73ba-4537-aa8a-47b8159a58c6" containerID="de87f3fa0926cd55169892f5dca7b2a6f75772887ada5b42ce946e5aa2529d3c" exitCode=0
Nov 03 20:32:10.698548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.698430    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65" event={"ID":"78765763-73ba-4537-aa8a-47b8159a58c6","Type":"ContainerDied","Data":"de87f3fa0926cd55169892f5dca7b2a6f75772887ada5b42ce946e5aa2529d3c"}
Nov 03 20:32:10.698691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.698669    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"
Nov 03 20:32:10.717602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.711264    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65" event={"ID":"78765763-73ba-4537-aa8a-47b8159a58c6","Type":"ContainerDied","Data":"3e43577cec5d14274e99eadf2a47897c1f7c0eba2a96dbfe051c76af6ead70cb"}
Nov 03 20:32:10.740151 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.739776    5659 scope.go:122] "RemoveContainer" containerID="11fb6de88e3166bb3ed3332661c84de96f809feada507d5f5678faae9959baaa"
Nov 03 20:32:10.741970 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:10.741577    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11fb6de88e3166bb3ed3332661c84de96f809feada507d5f5678faae9959baaa\": container with ID starting with 11fb6de88e3166bb3ed3332661c84de96f809feada507d5f5678faae9959baaa not found: ID does not exist" containerID="11fb6de88e3166bb3ed3332661c84de96f809feada507d5f5678faae9959baaa"
Nov 03 20:32:10.741970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.741620    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11fb6de88e3166bb3ed3332661c84de96f809feada507d5f5678faae9959baaa"} err="failed to get container status \"11fb6de88e3166bb3ed3332661c84de96f809feada507d5f5678faae9959baaa\": rpc error: code = NotFound desc = could not find container \"11fb6de88e3166bb3ed3332661c84de96f809feada507d5f5678faae9959baaa\": container with ID starting with 11fb6de88e3166bb3ed3332661c84de96f809feada507d5f5678faae9959baaa not found: ID does not exist"
Nov 03 20:32:10.741970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.741645    5659 scope.go:122] "RemoveContainer" containerID="de87f3fa0926cd55169892f5dca7b2a6f75772887ada5b42ce946e5aa2529d3c"
Nov 03 20:32:10.773783 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.764852    5659 scope.go:122] "RemoveContainer" containerID="de87f3fa0926cd55169892f5dca7b2a6f75772887ada5b42ce946e5aa2529d3c"
Nov 03 20:32:10.774192 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:10.774160    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de87f3fa0926cd55169892f5dca7b2a6f75772887ada5b42ce946e5aa2529d3c\": container with ID starting with de87f3fa0926cd55169892f5dca7b2a6f75772887ada5b42ce946e5aa2529d3c not found: ID does not exist" containerID="de87f3fa0926cd55169892f5dca7b2a6f75772887ada5b42ce946e5aa2529d3c"
Nov 03 20:32:10.774389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.774360    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de87f3fa0926cd55169892f5dca7b2a6f75772887ada5b42ce946e5aa2529d3c"} err="failed to get container status \"de87f3fa0926cd55169892f5dca7b2a6f75772887ada5b42ce946e5aa2529d3c\": rpc error: code = NotFound desc = could not find container \"de87f3fa0926cd55169892f5dca7b2a6f75772887ada5b42ce946e5aa2529d3c\": container with ID starting with de87f3fa0926cd55169892f5dca7b2a6f75772887ada5b42ce946e5aa2529d3c not found: ID does not exist"
Nov 03 20:32:10.822505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.821579    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"]
Nov 03 20:32:10.852578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.852503    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"]
Nov 03 20:32:10.863102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.863032    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2b5703-7c44b968cb-rgr65"]
Nov 03 20:32:10.873630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.873572    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"]
Nov 03 20:32:10.880813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.880787    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c557ghsgt"]
Nov 03 20:32:10.930591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.930549    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"]
Nov 03 20:32:10.937471 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:32:10.937044    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76f8240_ab05_410c_8a60_1d7b8cdde912.slice/crio-3dc1eff39848d7307d10c135b94e8283bc50d6fa9396a60e9dad7eefe689129b WatchSource:0}: Error finding container 3dc1eff39848d7307d10c135b94e8283bc50d6fa9396a60e9dad7eefe689129b: Status 404 returned error can't find the container with id 3dc1eff39848d7307d10c135b94e8283bc50d6fa9396a60e9dad7eefe689129b
Nov 03 20:32:10.948532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.940836    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:32:10.948532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.940925    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:32:10.951920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.951686    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:10.958227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:10.958197    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4ctkr"
Nov 03 20:32:11.078157 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:32:11.078095    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece7bfca_bf0e_4422_ad06_1df2ca9a08bf.slice/crio-3143c7233bd7be1365e94e6d29d30ae0d028b7ccc505a5fd7d4fa27376fd9b24 WatchSource:0}: Error finding container 3143c7233bd7be1365e94e6d29d30ae0d028b7ccc505a5fd7d4fa27376fd9b24: Status 404 returned error can't find the container with id 3143c7233bd7be1365e94e6d29d30ae0d028b7ccc505a5fd7d4fa27376fd9b24
Nov 03 20:32:11.086169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.086146    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jhb7v"]
Nov 03 20:32:11.226295 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.226261    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"
Nov 03 20:32:11.265779 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.265742    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-m2pgc"]
Nov 03 20:32:11.331241 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.330548    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"]
Nov 03 20:32:11.342852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.332535    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bc8b2605-2520-455c-aa28-6d81b5601ab8" containerName="revoke-keys"
Nov 03 20:32:11.342852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.332575    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8b2605-2520-455c-aa28-6d81b5601ab8" containerName="revoke-keys"
Nov 03 20:32:11.342852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.332610    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bc8b2605-2520-455c-aa28-6d81b5601ab8" containerName="revoke-keys"
Nov 03 20:32:11.342852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.332618    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8b2605-2520-455c-aa28-6d81b5601ab8" containerName="revoke-keys"
Nov 03 20:32:11.342852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.333078    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8b2605-2520-455c-aa28-6d81b5601ab8" containerName="revoke-keys"
Nov 03 20:32:11.342852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.333118    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8b2605-2520-455c-aa28-6d81b5601ab8" containerName="revoke-keys"
Nov 03 20:32:11.351639 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.351615    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"
Nov 03 20:32:11.362027 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.362003    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"]
Nov 03 20:32:11.377572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.377535    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f7b87d695-dwgqt"]
Nov 03 20:32:11.408771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.408697    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-xks-ca\") pod \"bc8b2605-2520-455c-aa28-6d81b5601ab8\" (UID: \"bc8b2605-2520-455c-aa28-6d81b5601ab8\") "
Nov 03 20:32:11.409060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.409035    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-openshift-ca\") pod \"bc8b2605-2520-455c-aa28-6d81b5601ab8\" (UID: \"bc8b2605-2520-455c-aa28-6d81b5601ab8\") "
Nov 03 20:32:11.409350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.409328    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8b2605-2520-455c-aa28-6d81b5601ab8-kube-api-access-6xzsd\" (UniqueName: \"kubernetes.io/projected/bc8b2605-2520-455c-aa28-6d81b5601ab8-kube-api-access-6xzsd\") pod \"bc8b2605-2520-455c-aa28-6d81b5601ab8\" (UID: \"bc8b2605-2520-455c-aa28-6d81b5601ab8\") "
Nov 03 20:32:11.411277 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.411254    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-openshift-ca" pod "bc8b2605-2520-455c-aa28-6d81b5601ab8" (UID: "bc8b2605-2520-455c-aa28-6d81b5601ab8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:11.417493 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.417453    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8b2605-2520-455c-aa28-6d81b5601ab8-kube-api-access-6xzsd" pod "bc8b2605-2520-455c-aa28-6d81b5601ab8" (UID: "bc8b2605-2520-455c-aa28-6d81b5601ab8"). InnerVolumeSpecName "kube-api-access-6xzsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:11.417675 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.417633    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-xks-ca" pod "bc8b2605-2520-455c-aa28-6d81b5601ab8" (UID: "bc8b2605-2520-455c-aa28-6d81b5601ab8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:11.512954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.511792    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-9zhhj\" (UID: \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"
Nov 03 20:32:11.512954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.511974    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-9zhhj\" (UID: \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"
Nov 03 20:32:11.512954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.512200    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glq5v\" (UniqueName: \"kubernetes.io/projected/7e9cc9f8-9bd0-4025-adcb-879bebd234af-kube-api-access-glq5v\") pod \"maas-api-revoke-sub-5bac6f1e-9zhhj\" (UID: \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"
Nov 03 20:32:11.512954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.512499    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:11.512954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.512522    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc8b2605-2520-455c-aa28-6d81b5601ab8-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:11.512954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.512538    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xzsd\" (UniqueName: \"kubernetes.io/projected/bc8b2605-2520-455c-aa28-6d81b5601ab8-kube-api-access-6xzsd\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:11.545357 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.545318    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-4ctkr"]
Nov 03 20:32:11.545711 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:32:11.545679    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd7fddd_b34d_4947_80fc_b48f082004c8.slice/crio-de0c6f605fbdbc487e679350c4fa394b64b315dae8fd51fb0ccadb92e159876c WatchSource:0}: Error finding container de0c6f605fbdbc487e679350c4fa394b64b315dae8fd51fb0ccadb92e159876c: Status 404 returned error can't find the container with id de0c6f605fbdbc487e679350c4fa394b64b315dae8fd51fb0ccadb92e159876c
Nov 03 20:32:11.615338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.613764    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-9zhhj\" (UID: \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"
Nov 03 20:32:11.615338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.613891    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-9zhhj\" (UID: \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"
Nov 03 20:32:11.615338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.614069    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glq5v\" (UniqueName: \"kubernetes.io/projected/7e9cc9f8-9bd0-4025-adcb-879bebd234af-kube-api-access-glq5v\") pod \"maas-api-revoke-sub-5bac6f1e-9zhhj\" (UID: \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"
Nov 03 20:32:11.618497 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.618286    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-9zhhj\" (UID: \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"
Nov 03 20:32:11.619043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.618992    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-9zhhj\" (UID: \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"
Nov 03 20:32:11.645808 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.645783    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glq5v\" (UniqueName: \"kubernetes.io/projected/7e9cc9f8-9bd0-4025-adcb-879bebd234af-kube-api-access-glq5v\") pod \"maas-api-revoke-sub-5bac6f1e-9zhhj\" (UID: \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"
Nov 03 20:32:11.681516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.681330    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"
Nov 03 20:32:11.777145 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.777088    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-m2pgc" event={"ID":"114f0567-5268-473e-aec5-50869958548c","Type":"ContainerStarted","Data":"0e1c81662122b02d657968bbf630674c89cae5b3ab5051384131e27790973a46"}
Nov 03 20:32:11.784653 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.783376    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4ctkr" event={"ID":"dcd7fddd-b34d-4947-80fc-b48f082004c8","Type":"ContainerStarted","Data":"0b418d4dea42aae21ee2a8bf864ea87208c59e9ae9fe8a5e51d6d08387bd1e45"}
Nov 03 20:32:11.784653 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.783431    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4ctkr" event={"ID":"dcd7fddd-b34d-4947-80fc-b48f082004c8","Type":"ContainerStarted","Data":"de0c6f605fbdbc487e679350c4fa394b64b315dae8fd51fb0ccadb92e159876c"}
Nov 03 20:32:11.787310 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.787283    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jhb7v" event={"ID":"ece7bfca-bf0e-4422-ad06-1df2ca9a08bf","Type":"ContainerStarted","Data":"3143c7233bd7be1365e94e6d29d30ae0d028b7ccc505a5fd7d4fa27376fd9b24"}
Nov 03 20:32:11.789991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.789971    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r" event={"ID":"f76f8240-ab05-410c-8a60-1d7b8cdde912","Type":"ContainerStarted","Data":"967f7fd677ea9e5d84ce8df0fd78ecbf8199f16e64bd2c017a91c1ba2a59b49b"}
Nov 03 20:32:11.790123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.790110    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r" event={"ID":"f76f8240-ab05-410c-8a60-1d7b8cdde912","Type":"ContainerStarted","Data":"3dc1eff39848d7307d10c135b94e8283bc50d6fa9396a60e9dad7eefe689129b"}
Nov 03 20:32:11.790858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.790843    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:11.801837 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.801819    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"
Nov 03 20:32:11.803053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.801982    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c" event={"ID":"bc8b2605-2520-455c-aa28-6d81b5601ab8","Type":"ContainerDied","Data":"b6d9455661da9d8ace67c77f62533819f3aa88b913fe85091fc66711469fb45a"}
Nov 03 20:32:11.803260 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.803223    5659 scope.go:122] "RemoveContainer" containerID="5b150933485f88c93209842e1d211cde05aee383bd2e107020685169fb715b35"
Nov 03 20:32:11.834449 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.834412    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f7b87d695-dwgqt" event={"ID":"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89","Type":"ContainerStarted","Data":"c75faa8c0ba1c552755a6439be5349926ba1b4cb5af66426374a9f82702df35a"}
Nov 03 20:32:11.835558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.835531    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f7b87d695-dwgqt" event={"ID":"d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89","Type":"ContainerStarted","Data":"6659951a890c012394ec9f600ab71500d51fba20a2d355dbda4e537e9d9fb52a"}
Nov 03 20:32:11.872473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.872440    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:11.876978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.876931    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-4ctkr" podStartSLOduration=8.876912613 podStartE2EDuration="8.876912613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:11.819651074 +0000 UTC m=+3283.974553493" watchObservedRunningTime="2026-11-03 20:32:11.876912613 +0000 UTC m=+3284.031815033"
Nov 03 20:32:11.879769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.879724    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r" podStartSLOduration=8.879706122 podStartE2EDuration="8.879706122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:11.858858489 +0000 UTC m=+3284.013760911" watchObservedRunningTime="2026-11-03 20:32:11.879706122 +0000 UTC m=+3284.034608543"
Nov 03 20:32:11.902663 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.901209    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-f7b87d695-dwgqt" podStartSLOduration=8.901193201 podStartE2EDuration="8.901193201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:11.897192758 +0000 UTC m=+3284.052095178" watchObservedRunningTime="2026-11-03 20:32:11.901193201 +0000 UTC m=+3284.056095619"
Nov 03 20:32:11.992908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:11.988251    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:12.152768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.151899    5659 scope.go:122] "RemoveContainer" containerID="46803512e5fdca6f59bd1a341348adc3e071097e2c1d345a67663226f2e58680"
Nov 03 20:32:12.298689 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.279336    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e0fa848-0fe6-41d8-b068-8bd42f81bb6e" path="/var/lib/kubelet/pods/0e0fa848-0fe6-41d8-b068-8bd42f81bb6e/volumes"
Nov 03 20:32:12.298689 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.280200    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a70e8c-6980-4dcb-aafd-0cf5feecd588" path="/var/lib/kubelet/pods/21a70e8c-6980-4dcb-aafd-0cf5feecd588/volumes"
Nov 03 20:32:12.298689 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.280827    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78765763-73ba-4537-aa8a-47b8159a58c6" path="/var/lib/kubelet/pods/78765763-73ba-4537-aa8a-47b8159a58c6/volumes"
Nov 03 20:32:12.312757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.312520    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"]
Nov 03 20:32:12.312757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.312571    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-thg6c"]
Nov 03 20:32:12.312757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.312673    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:12.457511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.451599    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"]
Nov 03 20:32:12.799507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.770621    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:12.918633 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.909393    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj" event={"ID":"7e9cc9f8-9bd0-4025-adcb-879bebd234af","Type":"ContainerStarted","Data":"5e14dc270f00189563786847f16bb9d44bb134355fa03b2cf58a40d41d4cf7ba"}
Nov 03 20:32:12.920169 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.920145    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:12.932737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.932012    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-f7b87d695-dwgqt"
Nov 03 20:32:12.942239 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.941149    5659 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2vf4r container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:32:12.942239 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.941196    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r" podUID="f76f8240-ab05-410c-8a60-1d7b8cdde912" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:32:12.977140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:12.976589    5659 scope.go:122] "RemoveContainer" containerID="f0abcc9f6027c84f772ede402f068a7ca290e16e3ed2ad866c80cff3249af30a"
Nov 03 20:32:13.382767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:13.382651    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:13.556272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:13.555171    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:13.892501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:13.874670    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:13.992298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:13.992186    5659 generic.go:356] "Generic (PLEG): container finished" podID="66914ad7-0ead-4525-8906-537ed24eda95" containerID="7ee7dfba12c886810519a9cb9a422d480b95324553fcbca9426e1c9c91a64b36" exitCode=22
Nov 03 20:32:13.995111 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:13.994962    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc8b2605-2520-455c-aa28-6d81b5601ab8" path="/var/lib/kubelet/pods/bc8b2605-2520-455c-aa28-6d81b5601ab8/volumes"
Nov 03 20:32:13.995558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:13.995531    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt" event={"ID":"66914ad7-0ead-4525-8906-537ed24eda95","Type":"ContainerDied","Data":"7ee7dfba12c886810519a9cb9a422d480b95324553fcbca9426e1c9c91a64b36"}
Nov 03 20:32:13.995660 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:13.995629    5659 scope.go:122] "RemoveContainer" containerID="46803512e5fdca6f59bd1a341348adc3e071097e2c1d345a67663226f2e58680"
Nov 03 20:32:13.996188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:13.996168    5659 scope.go:122] "RemoveContainer" containerID="7ee7dfba12c886810519a9cb9a422d480b95324553fcbca9426e1c9c91a64b36"
Nov 03 20:32:13.997001 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:13.996822    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-4bdd0b59-286lt_odh-ai-gateway-infra(66914ad7-0ead-4525-8906-537ed24eda95)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt" podUID="66914ad7-0ead-4525-8906-537ed24eda95"
Nov 03 20:32:14.004753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:14.004732    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6" event={"ID":"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20","Type":"ContainerStarted","Data":"21c7434244f7cf2322c411ec695bd6849be8ffee4cfe3b44ad108d43008cb0c1"}
Nov 03 20:32:14.010767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:14.010735    5659 scope.go:122] "RemoveContainer" containerID="d4d88d65990712c13be6304866dce1ae0b97fb311f8eeefe6a67f50c627d093d"
Nov 03 20:32:14.011180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:14.011155    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj" event={"ID":"7e9cc9f8-9bd0-4025-adcb-879bebd234af","Type":"ContainerStarted","Data":"d4d88d65990712c13be6304866dce1ae0b97fb311f8eeefe6a67f50c627d093d"}
Nov 03 20:32:14.868580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:14.860326    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:15.022539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.022498    5659 generic.go:356] "Generic (PLEG): container finished" podID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" containerID="21c7434244f7cf2322c411ec695bd6849be8ffee4cfe3b44ad108d43008cb0c1" exitCode=7
Nov 03 20:32:15.022539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.022527    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6" event={"ID":"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20","Type":"ContainerDied","Data":"21c7434244f7cf2322c411ec695bd6849be8ffee4cfe3b44ad108d43008cb0c1"}
Nov 03 20:32:15.022827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.022585    5659 scope.go:122] "RemoveContainer" containerID="f0abcc9f6027c84f772ede402f068a7ca290e16e3ed2ad866c80cff3249af30a"
Nov 03 20:32:15.023127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.023095    5659 scope.go:122] "RemoveContainer" containerID="21c7434244f7cf2322c411ec695bd6849be8ffee4cfe3b44ad108d43008cb0c1"
Nov 03 20:32:15.023579 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:15.023552    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-vvgj6_odh-ai-gateway-infra(c9d6fc0d-cefc-4d63-bc06-70e3b394ab20)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6" podUID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20"
Nov 03 20:32:15.024564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.024540    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-m2pgc" event={"ID":"114f0567-5268-473e-aec5-50869958548c","Type":"ContainerStarted","Data":"1e29a2bd83eb5e614f73fa36071c339450a758d70656f716a4c82e679e815dc2"}
Nov 03 20:32:15.026832 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.026808    5659 generic.go:356] "Generic (PLEG): container finished" podID="7e9cc9f8-9bd0-4025-adcb-879bebd234af" containerID="d4d88d65990712c13be6304866dce1ae0b97fb311f8eeefe6a67f50c627d093d" exitCode=22
Nov 03 20:32:15.026926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.026886    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj" event={"ID":"7e9cc9f8-9bd0-4025-adcb-879bebd234af","Type":"ContainerDied","Data":"d4d88d65990712c13be6304866dce1ae0b97fb311f8eeefe6a67f50c627d093d"}
Nov 03 20:32:15.026981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.026930    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj" event={"ID":"7e9cc9f8-9bd0-4025-adcb-879bebd234af","Type":"ContainerStarted","Data":"a08bc2b2bdfae4993c590ce3598131b3042328ba2d0e63ff8fdf975a93083535"}
Nov 03 20:32:15.029009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.028985    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jhb7v" event={"ID":"ece7bfca-bf0e-4422-ad06-1df2ca9a08bf","Type":"ContainerStarted","Data":"f7dfcc0144d378b8dd9b3f380dbdbd6a841347ecc814f439a9e42d8ffb9c78e3"}
Nov 03 20:32:15.037893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.037851    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"]
Nov 03 20:32:15.107673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.098367    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"]
Nov 03 20:32:15.116904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.114306    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jhb7v" podStartSLOduration=8.762491256 podStartE2EDuration="12.114291972s" totalImagesPullingTime="3.351800716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:15.107219981 +0000 UTC m=+3287.262122403" watchObservedRunningTime="2026-11-03 20:32:15.114291972 +0000 UTC m=+3287.269194390"
Nov 03 20:32:15.159859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.155456    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-m2pgc" podStartSLOduration=9.762718482 podStartE2EDuration="13.155394967s" totalImagesPullingTime="3.392676485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:15.150690908 +0000 UTC m=+3287.305593329" watchObservedRunningTime="2026-11-03 20:32:15.155394967 +0000 UTC m=+3287.310297393"
Nov 03 20:32:15.246980 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:15.246805    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:32:15.283959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.283790    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj" podStartSLOduration=13.283768798 podStartE2EDuration="13.283768798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:15.282587353 +0000 UTC m=+3287.437489774" watchObservedRunningTime="2026-11-03 20:32:15.283768798 +0000 UTC m=+3287.438671217"
Nov 03 20:32:15.320815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.320781    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"
Nov 03 20:32:15.373224 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.373183    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"]
Nov 03 20:32:15.374731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.374703    5659 cpu_manager.go:414] "containerMap: removing container" podUID="66914ad7-0ead-4525-8906-537ed24eda95" containerName="revoke-keys"
Nov 03 20:32:15.374862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.374736    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66914ad7-0ead-4525-8906-537ed24eda95" containerName="revoke-keys"
Nov 03 20:32:15.374862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.374764    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bc8b2605-2520-455c-aa28-6d81b5601ab8" containerName="revoke-keys"
Nov 03 20:32:15.374862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.374773    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8b2605-2520-455c-aa28-6d81b5601ab8" containerName="revoke-keys"
Nov 03 20:32:15.374862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.374856    5659 cpu_manager.go:414] "containerMap: removing container" podUID="66914ad7-0ead-4525-8906-537ed24eda95" containerName="revoke-keys"
Nov 03 20:32:15.374990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.374867    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66914ad7-0ead-4525-8906-537ed24eda95" containerName="revoke-keys"
Nov 03 20:32:15.375092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.375076    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="66914ad7-0ead-4525-8906-537ed24eda95" containerName="revoke-keys"
Nov 03 20:32:15.375155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.375102    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8b2605-2520-455c-aa28-6d81b5601ab8" containerName="revoke-keys"
Nov 03 20:32:15.375155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.375127    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="66914ad7-0ead-4525-8906-537ed24eda95" containerName="revoke-keys"
Nov 03 20:32:15.380035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.380008    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"
Nov 03 20:32:15.390839 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.390630    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"]
Nov 03 20:32:15.420731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.420680    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66914ad7-0ead-4525-8906-537ed24eda95-kube-api-access-2m54r\" (UniqueName: \"kubernetes.io/projected/66914ad7-0ead-4525-8906-537ed24eda95-kube-api-access-2m54r\") pod \"66914ad7-0ead-4525-8906-537ed24eda95\" (UID: \"66914ad7-0ead-4525-8906-537ed24eda95\") "
Nov 03 20:32:15.420932 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.420770    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-openshift-ca\") pod \"66914ad7-0ead-4525-8906-537ed24eda95\" (UID: \"66914ad7-0ead-4525-8906-537ed24eda95\") "
Nov 03 20:32:15.420932 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.420833    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-xks-ca\") pod \"66914ad7-0ead-4525-8906-537ed24eda95\" (UID: \"66914ad7-0ead-4525-8906-537ed24eda95\") "
Nov 03 20:32:15.421373 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.421320    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfrtd\" (UniqueName: \"kubernetes.io/projected/dc52ec5a-b374-423a-8045-b52cf4b88d5d-kube-api-access-xfrtd\") pod \"maas-api-revoke-sub-abad4031-2bhgb\" (UID: \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"
Nov 03 20:32:15.421628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.421593    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-2bhgb\" (UID: \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"
Nov 03 20:32:15.422356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.421656    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-2bhgb\" (UID: \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"
Nov 03 20:32:15.422356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.421797    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-openshift-ca" pod "66914ad7-0ead-4525-8906-537ed24eda95" (UID: "66914ad7-0ead-4525-8906-537ed24eda95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:15.424208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.424179    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-xks-ca" pod "66914ad7-0ead-4525-8906-537ed24eda95" (UID: "66914ad7-0ead-4525-8906-537ed24eda95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:15.425125 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.425107    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66914ad7-0ead-4525-8906-537ed24eda95-kube-api-access-2m54r" pod "66914ad7-0ead-4525-8906-537ed24eda95" (UID: "66914ad7-0ead-4525-8906-537ed24eda95"). InnerVolumeSpecName "kube-api-access-2m54r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:15.523937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.523899    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfrtd\" (UniqueName: \"kubernetes.io/projected/dc52ec5a-b374-423a-8045-b52cf4b88d5d-kube-api-access-xfrtd\") pod \"maas-api-revoke-sub-abad4031-2bhgb\" (UID: \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"
Nov 03 20:32:15.524133 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.524073    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-2bhgb\" (UID: \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"
Nov 03 20:32:15.524133 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.524103    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-2bhgb\" (UID: \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"
Nov 03 20:32:15.524319 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.524299    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m54r\" (UniqueName: \"kubernetes.io/projected/66914ad7-0ead-4525-8906-537ed24eda95-kube-api-access-2m54r\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:15.524392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.524324    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:15.524392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.524341    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66914ad7-0ead-4525-8906-537ed24eda95-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:15.524818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.524795    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abad4031-2bhgb\" (UID: \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"
Nov 03 20:32:15.526690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.526667    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abad4031-2bhgb\" (UID: \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"
Nov 03 20:32:15.538046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.537977    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfrtd\" (UniqueName: \"kubernetes.io/projected/dc52ec5a-b374-423a-8045-b52cf4b88d5d-kube-api-access-xfrtd\") pod \"maas-api-revoke-sub-abad4031-2bhgb\" (UID: \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"
Nov 03 20:32:15.724474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.724433    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"
Nov 03 20:32:15.919557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:15.919459    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:32:16.051232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.051187    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt" event={"ID":"66914ad7-0ead-4525-8906-537ed24eda95","Type":"ContainerDied","Data":"344a20779d2359c07eb95f7e6ab791deb00b107a90e048b92cc94e04d20c5359"}
Nov 03 20:32:16.051526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.051276    5659 scope.go:122] "RemoveContainer" containerID="7ee7dfba12c886810519a9cb9a422d480b95324553fcbca9426e1c9c91a64b36"
Nov 03 20:32:16.051526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.051380    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"
Nov 03 20:32:16.055294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.055262    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:16.063275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.063220    5659 generic.go:356] "Generic (PLEG): container finished" podID="7e9cc9f8-9bd0-4025-adcb-879bebd234af" containerID="a08bc2b2bdfae4993c590ce3598131b3042328ba2d0e63ff8fdf975a93083535" exitCode=22
Nov 03 20:32:16.090561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.080894    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj" event={"ID":"7e9cc9f8-9bd0-4025-adcb-879bebd234af","Type":"ContainerDied","Data":"a08bc2b2bdfae4993c590ce3598131b3042328ba2d0e63ff8fdf975a93083535"}
Nov 03 20:32:16.090561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.080986    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"]
Nov 03 20:32:16.090561 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.081683    5659 scope.go:122] "RemoveContainer" containerID="a08bc2b2bdfae4993c590ce3598131b3042328ba2d0e63ff8fdf975a93083535"
Nov 03 20:32:16.090561 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:16.085946    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-9zhhj_odh-ai-gateway-infra(7e9cc9f8-9bd0-4025-adcb-879bebd234af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj" podUID="7e9cc9f8-9bd0-4025-adcb-879bebd234af"
Nov 03 20:32:16.119440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.113710    5659 scope.go:122] "RemoveContainer" containerID="d4d88d65990712c13be6304866dce1ae0b97fb311f8eeefe6a67f50c627d093d"
Nov 03 20:32:16.209540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.209474    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"]
Nov 03 20:32:16.249142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.237101    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bdd0b59-286lt"]
Nov 03 20:32:16.372366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.372280    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"
Nov 03 20:32:16.407624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.407583    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"]
Nov 03 20:32:16.409125 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.409097    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" containerName="revoke-keys"
Nov 03 20:32:16.409289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.409276    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" containerName="revoke-keys"
Nov 03 20:32:16.409508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.409492    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" containerName="revoke-keys"
Nov 03 20:32:16.409617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.409597    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" containerName="revoke-keys"
Nov 03 20:32:16.409768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.409646    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" containerName="revoke-keys"
Nov 03 20:32:16.409768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.409653    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" containerName="revoke-keys"
Nov 03 20:32:16.409768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.409671    5659 cpu_manager.go:414] "containerMap: removing container" podUID="66914ad7-0ead-4525-8906-537ed24eda95" containerName="revoke-keys"
Nov 03 20:32:16.409768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.409678    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66914ad7-0ead-4525-8906-537ed24eda95" containerName="revoke-keys"
Nov 03 20:32:16.410017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.409900    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" containerName="revoke-keys"
Nov 03 20:32:16.410017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.409922    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" containerName="revoke-keys"
Nov 03 20:32:16.410017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.409933    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="66914ad7-0ead-4525-8906-537ed24eda95" containerName="revoke-keys"
Nov 03 20:32:16.414782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.414752    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"
Nov 03 20:32:16.427194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.427157    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"]
Nov 03 20:32:16.467265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.466454    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-xks-ca\") pod \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\" (UID: \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\") "
Nov 03 20:32:16.467265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.466664    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-kube-api-access-dctlq\" (UniqueName: \"kubernetes.io/projected/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-kube-api-access-dctlq\") pod \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\" (UID: \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\") "
Nov 03 20:32:16.467265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.466823    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-openshift-ca\") pod \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\" (UID: \"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20\") "
Nov 03 20:32:16.467620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.467347    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-9l7nl\" (UID: \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"
Nov 03 20:32:16.467620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.467568    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-9l7nl\" (UID: \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"
Nov 03 20:32:16.467620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.467605    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dts6b\" (UniqueName: \"kubernetes.io/projected/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-kube-api-access-dts6b\") pod \"maas-api-revoke-sub-403cf98d-9l7nl\" (UID: \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"
Nov 03 20:32:16.468143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.468046    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-openshift-ca" pod "c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" (UID: "c9d6fc0d-cefc-4d63-bc06-70e3b394ab20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:16.470648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.470617    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-xks-ca" pod "c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" (UID: "c9d6fc0d-cefc-4d63-bc06-70e3b394ab20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:16.472166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.472134    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-kube-api-access-dctlq" pod "c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" (UID: "c9d6fc0d-cefc-4d63-bc06-70e3b394ab20"). InnerVolumeSpecName "kube-api-access-dctlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:16.569297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.569260    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-9l7nl\" (UID: \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"
Nov 03 20:32:16.569552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.569316    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dts6b\" (UniqueName: \"kubernetes.io/projected/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-kube-api-access-dts6b\") pod \"maas-api-revoke-sub-403cf98d-9l7nl\" (UID: \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"
Nov 03 20:32:16.569629 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.569609    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-9l7nl\" (UID: \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"
Nov 03 20:32:16.569854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.569838    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:16.569922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.569861    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:16.569922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.569879    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dctlq\" (UniqueName: \"kubernetes.io/projected/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20-kube-api-access-dctlq\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:16.570397 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.570371    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-9l7nl\" (UID: \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"
Nov 03 20:32:16.571993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.571967    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-9l7nl\" (UID: \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"
Nov 03 20:32:16.582066 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.582035    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dts6b\" (UniqueName: \"kubernetes.io/projected/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-kube-api-access-dts6b\") pod \"maas-api-revoke-sub-403cf98d-9l7nl\" (UID: \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"
Nov 03 20:32:16.769377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.769342    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"
Nov 03 20:32:16.940404 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:16.940325    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:17.085647 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:17.084122    5659 scope.go:122] "RemoveContainer" containerID="a08bc2b2bdfae4993c590ce3598131b3042328ba2d0e63ff8fdf975a93083535"
Nov 03 20:32:17.085647 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:17.084840    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-9zhhj_odh-ai-gateway-infra(7e9cc9f8-9bd0-4025-adcb-879bebd234af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj" podUID="7e9cc9f8-9bd0-4025-adcb-879bebd234af"
Nov 03 20:32:17.086298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:17.086265    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb" event={"ID":"dc52ec5a-b374-423a-8045-b52cf4b88d5d","Type":"ContainerStarted","Data":"e1b88249beb81b27fff5b185886034ebfaab0d2ace7ed3be34393747920777cb"}
Nov 03 20:32:17.086376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:17.086330    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb" event={"ID":"dc52ec5a-b374-423a-8045-b52cf4b88d5d","Type":"ContainerStarted","Data":"a9756658a195b5a485c12b3350f7839849c403c0afbaeede2135aaf5af9fd087"}
Nov 03 20:32:17.089934 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:17.089902    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6" event={"ID":"c9d6fc0d-cefc-4d63-bc06-70e3b394ab20","Type":"ContainerDied","Data":"6dbbd7a4577a2664ff2ef49159687bcb11787fc82907c31789a866eef704a95f"}
Nov 03 20:32:17.090054 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:17.089956    5659 scope.go:122] "RemoveContainer" containerID="21c7434244f7cf2322c411ec695bd6849be8ffee4cfe3b44ad108d43008cb0c1"
Nov 03 20:32:17.090054 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:17.089915    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"
Nov 03 20:32:17.094228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:17.094013    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"]
Nov 03 20:32:17.135546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:17.135471    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb" podStartSLOduration=5.135450003 podStartE2EDuration="5.135450003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:17.12935668 +0000 UTC m=+3289.284259100" watchObservedRunningTime="2026-11-03 20:32:17.135450003 +0000 UTC m=+3289.290352424"
Nov 03 20:32:17.157669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:17.157637    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"]
Nov 03 20:32:17.168440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:17.168403    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-vvgj6"]
Nov 03 20:32:17.974918 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:17.974877    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66914ad7-0ead-4525-8906-537ed24eda95" path="/var/lib/kubelet/pods/66914ad7-0ead-4525-8906-537ed24eda95/volumes"
Nov 03 20:32:17.975554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:17.975536    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" path="/var/lib/kubelet/pods/c9d6fc0d-cefc-4d63-bc06-70e3b394ab20/volumes"
Nov 03 20:32:18.056167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:18.056116    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:18.125936 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:18.125881    5659 generic.go:356] "Generic (PLEG): container finished" podID="dc52ec5a-b374-423a-8045-b52cf4b88d5d" containerID="e1b88249beb81b27fff5b185886034ebfaab0d2ace7ed3be34393747920777cb" exitCode=22
Nov 03 20:32:18.126333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:18.126305    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb" event={"ID":"dc52ec5a-b374-423a-8045-b52cf4b88d5d","Type":"ContainerDied","Data":"e1b88249beb81b27fff5b185886034ebfaab0d2ace7ed3be34393747920777cb"}
Nov 03 20:32:18.127372 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:18.127264    5659 scope.go:122] "RemoveContainer" containerID="e1b88249beb81b27fff5b185886034ebfaab0d2ace7ed3be34393747920777cb"
Nov 03 20:32:18.131588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:18.129852    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl" event={"ID":"ba1342a5-53e9-4494-a78a-d495dfcf4a0f","Type":"ContainerStarted","Data":"f4154a817fda862f38493a9c01f9a4c560f6100be62e1e8c2b73afacfc0a3e37"}
Nov 03 20:32:18.131588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:18.129911    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl" event={"ID":"ba1342a5-53e9-4494-a78a-d495dfcf4a0f","Type":"ContainerStarted","Data":"8de4ea7d244c68a9c35c35167c0335bb997270c0161a6223c1a4d95b654cedcc"}
Nov 03 20:32:18.186310 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:18.186238    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl" podStartSLOduration=5.186214611 podStartE2EDuration="5.186214611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:18.178813834 +0000 UTC m=+3290.333716254" watchObservedRunningTime="2026-11-03 20:32:18.186214611 +0000 UTC m=+3290.341117031"
Nov 03 20:32:18.978510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:18.957084    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:19.179504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:19.178924    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb" event={"ID":"dc52ec5a-b374-423a-8045-b52cf4b88d5d","Type":"ContainerStarted","Data":"a434ba0e429464be5f0b93bdb98cc757413cc28de914b256aabb55d087d5487d"}
Nov 03 20:32:19.203505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:19.182303    5659 generic.go:356] "Generic (PLEG): container finished" podID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f" containerID="f4154a817fda862f38493a9c01f9a4c560f6100be62e1e8c2b73afacfc0a3e37" exitCode=22
Nov 03 20:32:19.203505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:19.182527    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl" event={"ID":"ba1342a5-53e9-4494-a78a-d495dfcf4a0f","Type":"ContainerDied","Data":"f4154a817fda862f38493a9c01f9a4c560f6100be62e1e8c2b73afacfc0a3e37"}
Nov 03 20:32:19.203505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:19.182910    5659 scope.go:122] "RemoveContainer" containerID="f4154a817fda862f38493a9c01f9a4c560f6100be62e1e8c2b73afacfc0a3e37"
Nov 03 20:32:19.783514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:19.768034    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:20.196376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:20.196335    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl" event={"ID":"ba1342a5-53e9-4494-a78a-d495dfcf4a0f","Type":"ContainerStarted","Data":"42eb6915f8f5400d8ad97a1dd7c28c4285871ffcf6e0bcc7d256108912a29de1"}
Nov 03 20:32:20.199015 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:20.198983    5659 generic.go:356] "Generic (PLEG): container finished" podID="dc52ec5a-b374-423a-8045-b52cf4b88d5d" containerID="a434ba0e429464be5f0b93bdb98cc757413cc28de914b256aabb55d087d5487d" exitCode=22
Nov 03 20:32:20.199138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:20.199053    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb" event={"ID":"dc52ec5a-b374-423a-8045-b52cf4b88d5d","Type":"ContainerDied","Data":"a434ba0e429464be5f0b93bdb98cc757413cc28de914b256aabb55d087d5487d"}
Nov 03 20:32:20.199138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:20.199107    5659 scope.go:122] "RemoveContainer" containerID="e1b88249beb81b27fff5b185886034ebfaab0d2ace7ed3be34393747920777cb"
Nov 03 20:32:20.199528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:20.199505    5659 scope.go:122] "RemoveContainer" containerID="a434ba0e429464be5f0b93bdb98cc757413cc28de914b256aabb55d087d5487d"
Nov 03 20:32:20.200653 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:20.199970    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-2bhgb_odh-ai-gateway-infra(dc52ec5a-b374-423a-8045-b52cf4b88d5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb" podUID="dc52ec5a-b374-423a-8045-b52cf4b88d5d"
Nov 03 20:32:20.682553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:20.682470    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:21.211865 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:21.211832    5659 generic.go:356] "Generic (PLEG): container finished" podID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f" containerID="42eb6915f8f5400d8ad97a1dd7c28c4285871ffcf6e0bcc7d256108912a29de1" exitCode=22
Nov 03 20:32:21.212384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:21.211926    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl" event={"ID":"ba1342a5-53e9-4494-a78a-d495dfcf4a0f","Type":"ContainerDied","Data":"42eb6915f8f5400d8ad97a1dd7c28c4285871ffcf6e0bcc7d256108912a29de1"}
Nov 03 20:32:21.212384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:21.212005    5659 scope.go:122] "RemoveContainer" containerID="f4154a817fda862f38493a9c01f9a4c560f6100be62e1e8c2b73afacfc0a3e37"
Nov 03 20:32:21.212384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:21.212340    5659 scope.go:122] "RemoveContainer" containerID="42eb6915f8f5400d8ad97a1dd7c28c4285871ffcf6e0bcc7d256108912a29de1"
Nov 03 20:32:21.212866 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:21.212837    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-9l7nl_odh-ai-gateway-infra(ba1342a5-53e9-4494-a78a-d495dfcf4a0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl" podUID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f"
Nov 03 20:32:21.214790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:21.214773    5659 scope.go:122] "RemoveContainer" containerID="a434ba0e429464be5f0b93bdb98cc757413cc28de914b256aabb55d087d5487d"
Nov 03 20:32:21.215144 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:21.215123    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-2bhgb_odh-ai-gateway-infra(dc52ec5a-b374-423a-8045-b52cf4b88d5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb" podUID="dc52ec5a-b374-423a-8045-b52cf4b88d5d"
Nov 03 20:32:21.700567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:21.699057    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:22.226818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:22.226789    5659 scope.go:122] "RemoveContainer" containerID="42eb6915f8f5400d8ad97a1dd7c28c4285871ffcf6e0bcc7d256108912a29de1"
Nov 03 20:32:22.227325 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:22.227185    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-9l7nl_odh-ai-gateway-infra(ba1342a5-53e9-4494-a78a-d495dfcf4a0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl" podUID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f"
Nov 03 20:32:22.357970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:22.357932    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:23.190309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:23.190264    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:23.514531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:23.514464    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:23.640341 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:23.640305    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:24.037538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:24.014630    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:24.856530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:24.854574    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:25.655505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:25.649009    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:26.179427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:26.179381    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:26.358888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:26.358847    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:26.498462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:26.498425    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:26.641647 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:26.641614    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:27.168511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:27.168375    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:27.968722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:27.968695    5659 scope.go:122] "RemoveContainer" containerID="a08bc2b2bdfae4993c590ce3598131b3042328ba2d0e63ff8fdf975a93083535"
Nov 03 20:32:28.053769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:28.053732    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:28.749320 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:28.749287    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:29.414732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:29.414643    5659 generic.go:356] "Generic (PLEG): container finished" podID="7e9cc9f8-9bd0-4025-adcb-879bebd234af" containerID="62ca5069dfb5959ec48baede222080f07e3e8375f866dd0f3d0e764e0df7b2a6" exitCode=22
Nov 03 20:32:29.414732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:29.414716    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj" event={"ID":"7e9cc9f8-9bd0-4025-adcb-879bebd234af","Type":"ContainerDied","Data":"62ca5069dfb5959ec48baede222080f07e3e8375f866dd0f3d0e764e0df7b2a6"}
Nov 03 20:32:29.415254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:29.414791    5659 scope.go:122] "RemoveContainer" containerID="a08bc2b2bdfae4993c590ce3598131b3042328ba2d0e63ff8fdf975a93083535"
Nov 03 20:32:29.415371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:29.415353    5659 scope.go:122] "RemoveContainer" containerID="62ca5069dfb5959ec48baede222080f07e3e8375f866dd0f3d0e764e0df7b2a6"
Nov 03 20:32:29.416050 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:29.415845    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-9zhhj_odh-ai-gateway-infra(7e9cc9f8-9bd0-4025-adcb-879bebd234af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj" podUID="7e9cc9f8-9bd0-4025-adcb-879bebd234af"
Nov 03 20:32:29.521321 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:29.521281    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:30.255099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.255057    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:30.451631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.451594    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"]
Nov 03 20:32:30.689926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.689900    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"
Nov 03 20:32:30.712243 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.712209    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-xks-ca\") pod \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\" (UID: \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\") "
Nov 03 20:32:30.712470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.712453    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-openshift-ca\") pod \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\" (UID: \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\") "
Nov 03 20:32:30.712636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.712614    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9cc9f8-9bd0-4025-adcb-879bebd234af-kube-api-access-glq5v\" (UniqueName: \"kubernetes.io/projected/7e9cc9f8-9bd0-4025-adcb-879bebd234af-kube-api-access-glq5v\") pod \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\" (UID: \"7e9cc9f8-9bd0-4025-adcb-879bebd234af\") "
Nov 03 20:32:30.714559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.714440    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-openshift-ca" pod "7e9cc9f8-9bd0-4025-adcb-879bebd234af" (UID: "7e9cc9f8-9bd0-4025-adcb-879bebd234af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:30.716655 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.716546    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9cc9f8-9bd0-4025-adcb-879bebd234af-kube-api-access-glq5v" pod "7e9cc9f8-9bd0-4025-adcb-879bebd234af" (UID: "7e9cc9f8-9bd0-4025-adcb-879bebd234af"). InnerVolumeSpecName "kube-api-access-glq5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:30.718240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.718214    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-xks-ca" pod "7e9cc9f8-9bd0-4025-adcb-879bebd234af" (UID: "7e9cc9f8-9bd0-4025-adcb-879bebd234af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:30.729713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.729663    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw"]
Nov 03 20:32:30.731082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.731053    5659 cpu_manager.go:414] "containerMap: removing container" podUID="7e9cc9f8-9bd0-4025-adcb-879bebd234af" containerName="revoke-keys"
Nov 03 20:32:30.731082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.731082    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9cc9f8-9bd0-4025-adcb-879bebd234af" containerName="revoke-keys"
Nov 03 20:32:30.731364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.731099    5659 cpu_manager.go:414] "containerMap: removing container" podUID="7e9cc9f8-9bd0-4025-adcb-879bebd234af" containerName="revoke-keys"
Nov 03 20:32:30.731364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.731107    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9cc9f8-9bd0-4025-adcb-879bebd234af" containerName="revoke-keys"
Nov 03 20:32:30.731364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.731135    5659 cpu_manager.go:414] "containerMap: removing container" podUID="7e9cc9f8-9bd0-4025-adcb-879bebd234af" containerName="revoke-keys"
Nov 03 20:32:30.731364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.731144    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9cc9f8-9bd0-4025-adcb-879bebd234af" containerName="revoke-keys"
Nov 03 20:32:30.731630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.731402    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9cc9f8-9bd0-4025-adcb-879bebd234af" containerName="revoke-keys"
Nov 03 20:32:30.731630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.731421    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9cc9f8-9bd0-4025-adcb-879bebd234af" containerName="revoke-keys"
Nov 03 20:32:30.731630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.731436    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9cc9f8-9bd0-4025-adcb-879bebd234af" containerName="revoke-keys"
Nov 03 20:32:30.731630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.731455    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d6fc0d-cefc-4d63-bc06-70e3b394ab20" containerName="revoke-keys"
Nov 03 20:32:30.740712 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.736781    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw"
Nov 03 20:32:30.764745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.763884    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 20:32:30.764745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.763998    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:32:30.764745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.764103    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-w5t25\""
Nov 03 20:32:30.772539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.769256    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw"]
Nov 03 20:32:30.815163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.815079    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn724\" (UniqueName: \"kubernetes.io/projected/28cd7a29-ef8f-4b85-b485-b636ae60d896-kube-api-access-qn724\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-57hcw\" (UID: \"28cd7a29-ef8f-4b85-b485-b636ae60d896\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw"
Nov 03 20:32:30.815163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.815125    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28cd7a29-ef8f-4b85-b485-b636ae60d896-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-57hcw\" (UID: \"28cd7a29-ef8f-4b85-b485-b636ae60d896\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw"
Nov 03 20:32:30.815385 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.815372    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:30.815427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.815387    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glq5v\" (UniqueName: \"kubernetes.io/projected/7e9cc9f8-9bd0-4025-adcb-879bebd234af-kube-api-access-glq5v\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:30.815427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.815398    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9cc9f8-9bd0-4025-adcb-879bebd234af-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:30.916963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.916928    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn724\" (UniqueName: \"kubernetes.io/projected/28cd7a29-ef8f-4b85-b485-b636ae60d896-kube-api-access-qn724\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-57hcw\" (UID: \"28cd7a29-ef8f-4b85-b485-b636ae60d896\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw"
Nov 03 20:32:30.916963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.916974    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28cd7a29-ef8f-4b85-b485-b636ae60d896-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-57hcw\" (UID: \"28cd7a29-ef8f-4b85-b485-b636ae60d896\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw"
Nov 03 20:32:30.917413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.917393    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28cd7a29-ef8f-4b85-b485-b636ae60d896-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-57hcw\" (UID: \"28cd7a29-ef8f-4b85-b485-b636ae60d896\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw"
Nov 03 20:32:30.929814 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:30.929786    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn724\" (UniqueName: \"kubernetes.io/projected/28cd7a29-ef8f-4b85-b485-b636ae60d896-kube-api-access-qn724\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-57hcw\" (UID: \"28cd7a29-ef8f-4b85-b485-b636ae60d896\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw"
Nov 03 20:32:31.076562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:31.072427    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:31.086524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:31.085325    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw"
Nov 03 20:32:31.432724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:31.432688    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw"]
Nov 03 20:32:31.433750 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:32:31.433720    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28cd7a29_ef8f_4b85_b485_b636ae60d896.slice/crio-57f2a083eb9c031a3c1eecab50e0ad9a57649a627f35de5201600647474004ae WatchSource:0}: Error finding container 57f2a083eb9c031a3c1eecab50e0ad9a57649a627f35de5201600647474004ae: Status 404 returned error can't find the container with id 57f2a083eb9c031a3c1eecab50e0ad9a57649a627f35de5201600647474004ae
Nov 03 20:32:31.438871 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:31.438842    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj" event={"ID":"7e9cc9f8-9bd0-4025-adcb-879bebd234af","Type":"ContainerDied","Data":"5e14dc270f00189563786847f16bb9d44bb134355fa03b2cf58a40d41d4cf7ba"}
Nov 03 20:32:31.438977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:31.438898    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"
Nov 03 20:32:31.438977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:31.438924    5659 scope.go:122] "RemoveContainer" containerID="62ca5069dfb5959ec48baede222080f07e3e8375f866dd0f3d0e764e0df7b2a6"
Nov 03 20:32:31.487912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:31.487866    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"]
Nov 03 20:32:31.492478 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:31.492448    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-9zhhj"]
Nov 03 20:32:31.846115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:31.846069    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:31.962671 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:31.962633    5659 scope.go:122] "RemoveContainer" containerID="a434ba0e429464be5f0b93bdb98cc757413cc28de914b256aabb55d087d5487d"
Nov 03 20:32:31.983767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:31.983732    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9cc9f8-9bd0-4025-adcb-879bebd234af" path="/var/lib/kubelet/pods/7e9cc9f8-9bd0-4025-adcb-879bebd234af/volumes"
Nov 03 20:32:32.473559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:32.473340    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw" event={"ID":"28cd7a29-ef8f-4b85-b485-b636ae60d896","Type":"ContainerStarted","Data":"57f2a083eb9c031a3c1eecab50e0ad9a57649a627f35de5201600647474004ae"}
Nov 03 20:32:32.754793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:32.754723    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:33.336520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:33.336464    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:33.533425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:33.533387    5659 generic.go:356] "Generic (PLEG): container finished" podID="dc52ec5a-b374-423a-8045-b52cf4b88d5d" containerID="381101bd9dcec9c77efaaf98df8cff6b0fc64430351691c35257a9f8df082960" exitCode=22
Nov 03 20:32:33.533795 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:33.533758    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb" event={"ID":"dc52ec5a-b374-423a-8045-b52cf4b88d5d","Type":"ContainerDied","Data":"381101bd9dcec9c77efaaf98df8cff6b0fc64430351691c35257a9f8df082960"}
Nov 03 20:32:33.533941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:33.533832    5659 scope.go:122] "RemoveContainer" containerID="a434ba0e429464be5f0b93bdb98cc757413cc28de914b256aabb55d087d5487d"
Nov 03 20:32:33.534364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:33.534346    5659 scope.go:122] "RemoveContainer" containerID="381101bd9dcec9c77efaaf98df8cff6b0fc64430351691c35257a9f8df082960"
Nov 03 20:32:33.534866 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:33.534839    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-abad4031-2bhgb_odh-ai-gateway-infra(dc52ec5a-b374-423a-8045-b52cf4b88d5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb" podUID="dc52ec5a-b374-423a-8045-b52cf4b88d5d"
Nov 03 20:32:33.965017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:33.964977    5659 scope.go:122] "RemoveContainer" containerID="42eb6915f8f5400d8ad97a1dd7c28c4285871ffcf6e0bcc7d256108912a29de1"
Nov 03 20:32:34.231607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.231373    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:34.399165 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.399122    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:34.585215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.585173    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"]
Nov 03 20:32:34.840493 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.840451    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"
Nov 03 20:32:34.886836 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.886743    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"]
Nov 03 20:32:34.888075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.888030    5659 cpu_manager.go:414] "containerMap: removing container" podUID="dc52ec5a-b374-423a-8045-b52cf4b88d5d" containerName="revoke-keys"
Nov 03 20:32:34.888075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.888057    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc52ec5a-b374-423a-8045-b52cf4b88d5d" containerName="revoke-keys"
Nov 03 20:32:34.888075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.888073    5659 cpu_manager.go:414] "containerMap: removing container" podUID="dc52ec5a-b374-423a-8045-b52cf4b88d5d" containerName="revoke-keys"
Nov 03 20:32:34.888075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.888081    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc52ec5a-b374-423a-8045-b52cf4b88d5d" containerName="revoke-keys"
Nov 03 20:32:34.888383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.888120    5659 cpu_manager.go:414] "containerMap: removing container" podUID="dc52ec5a-b374-423a-8045-b52cf4b88d5d" containerName="revoke-keys"
Nov 03 20:32:34.888383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.888128    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc52ec5a-b374-423a-8045-b52cf4b88d5d" containerName="revoke-keys"
Nov 03 20:32:34.888383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.888367    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc52ec5a-b374-423a-8045-b52cf4b88d5d" containerName="revoke-keys"
Nov 03 20:32:34.888383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.888384    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc52ec5a-b374-423a-8045-b52cf4b88d5d" containerName="revoke-keys"
Nov 03 20:32:34.888559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.888398    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc52ec5a-b374-423a-8045-b52cf4b88d5d" containerName="revoke-keys"
Nov 03 20:32:34.893316 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.893292    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"
Nov 03 20:32:34.922678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.919555    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"]
Nov 03 20:32:34.922678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.919971    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-xks-ca\") pod \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\" (UID: \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\") "
Nov 03 20:32:34.922678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.920032    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc52ec5a-b374-423a-8045-b52cf4b88d5d-kube-api-access-xfrtd\" (UniqueName: \"kubernetes.io/projected/dc52ec5a-b374-423a-8045-b52cf4b88d5d-kube-api-access-xfrtd\") pod \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\" (UID: \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\") "
Nov 03 20:32:34.922678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.920126    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-openshift-ca\") pod \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\" (UID: \"dc52ec5a-b374-423a-8045-b52cf4b88d5d\") "
Nov 03 20:32:34.922678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.920586    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-openshift-ca" pod "dc52ec5a-b374-423a-8045-b52cf4b88d5d" (UID: "dc52ec5a-b374-423a-8045-b52cf4b88d5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:34.922678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.921408    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:34.939613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.939548    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc52ec5a-b374-423a-8045-b52cf4b88d5d-kube-api-access-xfrtd" pod "dc52ec5a-b374-423a-8045-b52cf4b88d5d" (UID: "dc52ec5a-b374-423a-8045-b52cf4b88d5d"). InnerVolumeSpecName "kube-api-access-xfrtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:34.953864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.953818    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-xks-ca" pod "dc52ec5a-b374-423a-8045-b52cf4b88d5d" (UID: "dc52ec5a-b374-423a-8045-b52cf4b88d5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:34.967032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:34.966978    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:35.080930 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.080838    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-vtgk2\" (UID: \"f95421fb-9e1d-4bf8-a721-86323c623e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"
Nov 03 20:32:35.081110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.080941    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmrc\" (UniqueName: \"kubernetes.io/projected/f95421fb-9e1d-4bf8-a721-86323c623e5b-kube-api-access-gqmrc\") pod \"maas-api-revoke-sub-266d0609-vtgk2\" (UID: \"f95421fb-9e1d-4bf8-a721-86323c623e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"
Nov 03 20:32:35.081229 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.081210    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-vtgk2\" (UID: \"f95421fb-9e1d-4bf8-a721-86323c623e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"
Nov 03 20:32:35.081309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.081294    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc52ec5a-b374-423a-8045-b52cf4b88d5d-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:35.081371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.081314    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfrtd\" (UniqueName: \"kubernetes.io/projected/dc52ec5a-b374-423a-8045-b52cf4b88d5d-kube-api-access-xfrtd\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:35.184351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.183025    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-vtgk2\" (UID: \"f95421fb-9e1d-4bf8-a721-86323c623e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"
Nov 03 20:32:35.184351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.183107    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-vtgk2\" (UID: \"f95421fb-9e1d-4bf8-a721-86323c623e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"
Nov 03 20:32:35.184351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.183253    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmrc\" (UniqueName: \"kubernetes.io/projected/f95421fb-9e1d-4bf8-a721-86323c623e5b-kube-api-access-gqmrc\") pod \"maas-api-revoke-sub-266d0609-vtgk2\" (UID: \"f95421fb-9e1d-4bf8-a721-86323c623e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"
Nov 03 20:32:35.185093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.185061    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-266d0609-vtgk2\" (UID: \"f95421fb-9e1d-4bf8-a721-86323c623e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"
Nov 03 20:32:35.187806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.187776    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-266d0609-vtgk2\" (UID: \"f95421fb-9e1d-4bf8-a721-86323c623e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"
Nov 03 20:32:35.200034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.200002    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmrc\" (UniqueName: \"kubernetes.io/projected/f95421fb-9e1d-4bf8-a721-86323c623e5b-kube-api-access-gqmrc\") pod \"maas-api-revoke-sub-266d0609-vtgk2\" (UID: \"f95421fb-9e1d-4bf8-a721-86323c623e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"
Nov 03 20:32:35.224898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.224864    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"
Nov 03 20:32:35.602521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.599879    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"]
Nov 03 20:32:35.609749 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.609710    5659 generic.go:356] "Generic (PLEG): container finished" podID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f" containerID="d8b5df56cdb0ba595d9d0bc4f626e96091804dc7ecf2814005367fa315d66f43" exitCode=22
Nov 03 20:32:35.609847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.609800    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl" event={"ID":"ba1342a5-53e9-4494-a78a-d495dfcf4a0f","Type":"ContainerDied","Data":"d8b5df56cdb0ba595d9d0bc4f626e96091804dc7ecf2814005367fa315d66f43"}
Nov 03 20:32:35.609886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.609848    5659 scope.go:122] "RemoveContainer" containerID="42eb6915f8f5400d8ad97a1dd7c28c4285871ffcf6e0bcc7d256108912a29de1"
Nov 03 20:32:35.613816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.613793    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw" event={"ID":"28cd7a29-ef8f-4b85-b485-b636ae60d896","Type":"ContainerStarted","Data":"849861399e18bab9aa6e3bac2fd37c8187a1c4326a401692d1f45906abee77eb"}
Nov 03 20:32:35.616160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.616132    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb" event={"ID":"dc52ec5a-b374-423a-8045-b52cf4b88d5d","Type":"ContainerDied","Data":"a9756658a195b5a485c12b3350f7839849c403c0afbaeede2135aaf5af9fd087"}
Nov 03 20:32:35.616253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.616226    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"
Nov 03 20:32:35.617618 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.617592    5659 scope.go:122] "RemoveContainer" containerID="d8b5df56cdb0ba595d9d0bc4f626e96091804dc7ecf2814005367fa315d66f43"
Nov 03 20:32:35.618076 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:35.618036    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-9l7nl_odh-ai-gateway-infra(ba1342a5-53e9-4494-a78a-d495dfcf4a0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl" podUID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f"
Nov 03 20:32:35.638709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.638682    5659 scope.go:122] "RemoveContainer" containerID="381101bd9dcec9c77efaaf98df8cff6b0fc64430351691c35257a9f8df082960"
Nov 03 20:32:35.644345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.644016    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:35.715510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.706618    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"]
Nov 03 20:32:35.798990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.795301    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abad4031-2bhgb"]
Nov 03 20:32:35.798990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.797244    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-57hcw" podStartSLOduration=29.939676201 podStartE2EDuration="33.797231172s" totalImagesPullingTime="3.857554971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:35.697147656 +0000 UTC m=+3307.852050077" watchObservedRunningTime="2026-11-03 20:32:35.797231172 +0000 UTC m=+3307.952133590"
Nov 03 20:32:35.979744 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:35.979699    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc52ec5a-b374-423a-8045-b52cf4b88d5d" path="/var/lib/kubelet/pods/dc52ec5a-b374-423a-8045-b52cf4b88d5d/volumes"
Nov 03 20:32:36.445457 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:36.445349    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:36.628551 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:36.628513    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2" event={"ID":"f95421fb-9e1d-4bf8-a721-86323c623e5b","Type":"ContainerStarted","Data":"b6bb457dd8d431e105f412b891cc5a5f8eea090b2dbb127fd24766dd1742583d"}
Nov 03 20:32:36.628755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:36.628571    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2" event={"ID":"f95421fb-9e1d-4bf8-a721-86323c623e5b","Type":"ContainerStarted","Data":"28cf7d39b1c8a3eb8d62a30dc1f511b324cf4db2ff9a31394e97bea4664d75c9"}
Nov 03 20:32:36.653409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:36.653366    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"]
Nov 03 20:32:36.657943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:36.657822    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2" podStartSLOduration=20.657803529 podStartE2EDuration="20.657803529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:36.648686911 +0000 UTC m=+3308.803589338" watchObservedRunningTime="2026-11-03 20:32:36.657803529 +0000 UTC m=+3308.812705950"
Nov 03 20:32:36.983672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:36.983645    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"
Nov 03 20:32:37.017517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.017413    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"]
Nov 03 20:32:37.018383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.018346    5659 cpu_manager.go:414] "containerMap: removing container" podUID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f" containerName="revoke-keys"
Nov 03 20:32:37.018383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.018370    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f" containerName="revoke-keys"
Nov 03 20:32:37.018599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.018471    5659 cpu_manager.go:414] "containerMap: removing container" podUID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f" containerName="revoke-keys"
Nov 03 20:32:37.018599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.018496    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f" containerName="revoke-keys"
Nov 03 20:32:37.018973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.018700    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f" containerName="revoke-keys"
Nov 03 20:32:37.018973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.018720    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f" containerName="revoke-keys"
Nov 03 20:32:37.022870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.022840    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"
Nov 03 20:32:37.054545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.052857    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"]
Nov 03 20:32:37.065507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.059927    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-openshift-ca\") pod \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\" (UID: \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\") "
Nov 03 20:32:37.065507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.059986    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-xks-ca\") pod \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\" (UID: \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\") "
Nov 03 20:32:37.065507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.060146    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-kube-api-access-dts6b\" (UniqueName: \"kubernetes.io/projected/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-kube-api-access-dts6b\") pod \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\" (UID: \"ba1342a5-53e9-4494-a78a-d495dfcf4a0f\") "
Nov 03 20:32:37.065507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.060801    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-xtzpl\" (UID: \"6b9cc76e-d155-4572-b544-15613ed861e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"
Nov 03 20:32:37.065507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.060998    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcdbq\" (UniqueName: \"kubernetes.io/projected/6b9cc76e-d155-4572-b544-15613ed861e0-kube-api-access-wcdbq\") pod \"maas-api-revoke-sub-5bac6f1e-xtzpl\" (UID: \"6b9cc76e-d155-4572-b544-15613ed861e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"
Nov 03 20:32:37.065507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.061240    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-xtzpl\" (UID: \"6b9cc76e-d155-4572-b544-15613ed861e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"
Nov 03 20:32:37.065507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.062022    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-openshift-ca" pod "ba1342a5-53e9-4494-a78a-d495dfcf4a0f" (UID: "ba1342a5-53e9-4494-a78a-d495dfcf4a0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:37.075538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.070623    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-xks-ca" pod "ba1342a5-53e9-4494-a78a-d495dfcf4a0f" (UID: "ba1342a5-53e9-4494-a78a-d495dfcf4a0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:37.075538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.071157    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-kube-api-access-dts6b" pod "ba1342a5-53e9-4494-a78a-d495dfcf4a0f" (UID: "ba1342a5-53e9-4494-a78a-d495dfcf4a0f"). InnerVolumeSpecName "kube-api-access-dts6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:37.163370 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.163321    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-xtzpl\" (UID: \"6b9cc76e-d155-4572-b544-15613ed861e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"
Nov 03 20:32:37.163599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.163428    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-xtzpl\" (UID: \"6b9cc76e-d155-4572-b544-15613ed861e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"
Nov 03 20:32:37.163599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.163560    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcdbq\" (UniqueName: \"kubernetes.io/projected/6b9cc76e-d155-4572-b544-15613ed861e0-kube-api-access-wcdbq\") pod \"maas-api-revoke-sub-5bac6f1e-xtzpl\" (UID: \"6b9cc76e-d155-4572-b544-15613ed861e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"
Nov 03 20:32:37.163746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.163639    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:37.163746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.163651    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:37.163746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.163662    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dts6b\" (UniqueName: \"kubernetes.io/projected/ba1342a5-53e9-4494-a78a-d495dfcf4a0f-kube-api-access-dts6b\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:37.164514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.164451    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac6f1e-xtzpl\" (UID: \"6b9cc76e-d155-4572-b544-15613ed861e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"
Nov 03 20:32:37.166920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.166895    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac6f1e-xtzpl\" (UID: \"6b9cc76e-d155-4572-b544-15613ed861e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"
Nov 03 20:32:37.176267 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.176227    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcdbq\" (UniqueName: \"kubernetes.io/projected/6b9cc76e-d155-4572-b544-15613ed861e0-kube-api-access-wcdbq\") pod \"maas-api-revoke-sub-5bac6f1e-xtzpl\" (UID: \"6b9cc76e-d155-4572-b544-15613ed861e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"
Nov 03 20:32:37.330583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.324640    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:37.372928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.371588    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"
Nov 03 20:32:37.655766 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.655674    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"
Nov 03 20:32:37.655766 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.655716    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl" event={"ID":"ba1342a5-53e9-4494-a78a-d495dfcf4a0f","Type":"ContainerDied","Data":"8de4ea7d244c68a9c35c35167c0335bb997270c0161a6223c1a4d95b654cedcc"}
Nov 03 20:32:37.656021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.655776    5659 scope.go:122] "RemoveContainer" containerID="d8b5df56cdb0ba595d9d0bc4f626e96091804dc7ecf2814005367fa315d66f43"
Nov 03 20:32:37.671501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.662633    5659 generic.go:356] "Generic (PLEG): container finished" podID="f95421fb-9e1d-4bf8-a721-86323c623e5b" containerID="b6bb457dd8d431e105f412b891cc5a5f8eea090b2dbb127fd24766dd1742583d" exitCode=22
Nov 03 20:32:37.671501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.663121    5659 scope.go:122] "RemoveContainer" containerID="b6bb457dd8d431e105f412b891cc5a5f8eea090b2dbb127fd24766dd1742583d"
Nov 03 20:32:37.671501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.663142    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2" event={"ID":"f95421fb-9e1d-4bf8-a721-86323c623e5b","Type":"ContainerDied","Data":"b6bb457dd8d431e105f412b891cc5a5f8eea090b2dbb127fd24766dd1742583d"}
Nov 03 20:32:37.719413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.719380    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"]
Nov 03 20:32:37.727092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.727064    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-9l7nl"]
Nov 03 20:32:37.755554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.755347    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"]
Nov 03 20:32:37.761831 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:32:37.761797    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9cc76e_d155_4572_b544_15613ed861e0.slice/crio-22af37dd16edfb3c46f4175c6a49e149e3b3055cc31a538b9a6bbdfe9d7168b2 WatchSource:0}: Error finding container 22af37dd16edfb3c46f4175c6a49e149e3b3055cc31a538b9a6bbdfe9d7168b2: Status 404 returned error can't find the container with id 22af37dd16edfb3c46f4175c6a49e149e3b3055cc31a538b9a6bbdfe9d7168b2
Nov 03 20:32:37.976825 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:37.976789    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f" path="/var/lib/kubelet/pods/ba1342a5-53e9-4494-a78a-d495dfcf4a0f/volumes"
Nov 03 20:32:38.332502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:38.326573    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:38.681965 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:38.681858    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2" event={"ID":"f95421fb-9e1d-4bf8-a721-86323c623e5b","Type":"ContainerStarted","Data":"c7a6a4518c24e36ee36b7ea1227ec1220716596688b2143524afae7245cc6805"}
Nov 03 20:32:38.690230 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:38.690191    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl" event={"ID":"6b9cc76e-d155-4572-b544-15613ed861e0","Type":"ContainerStarted","Data":"903983ca48be4d583f6a0295688ebf00cf0e2429d0d9cab95ea88102a372bceb"}
Nov 03 20:32:38.690534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:38.690513    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl" event={"ID":"6b9cc76e-d155-4572-b544-15613ed861e0","Type":"ContainerStarted","Data":"22af37dd16edfb3c46f4175c6a49e149e3b3055cc31a538b9a6bbdfe9d7168b2"}
Nov 03 20:32:38.734599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:38.734540    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl" podStartSLOduration=6.734524385 podStartE2EDuration="6.734524385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:38.730123936 +0000 UTC m=+3310.885026357" watchObservedRunningTime="2026-11-03 20:32:38.734524385 +0000 UTC m=+3310.889426803"
Nov 03 20:32:39.327740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:39.327701    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:39.701794 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:39.701695    5659 generic.go:356] "Generic (PLEG): container finished" podID="6b9cc76e-d155-4572-b544-15613ed861e0" containerID="903983ca48be4d583f6a0295688ebf00cf0e2429d0d9cab95ea88102a372bceb" exitCode=22
Nov 03 20:32:39.702231 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:39.701790    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl" event={"ID":"6b9cc76e-d155-4572-b544-15613ed861e0","Type":"ContainerDied","Data":"903983ca48be4d583f6a0295688ebf00cf0e2429d0d9cab95ea88102a372bceb"}
Nov 03 20:32:39.702231 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:39.702152    5659 scope.go:122] "RemoveContainer" containerID="903983ca48be4d583f6a0295688ebf00cf0e2429d0d9cab95ea88102a372bceb"
Nov 03 20:32:39.704371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:39.704257    5659 generic.go:356] "Generic (PLEG): container finished" podID="f95421fb-9e1d-4bf8-a721-86323c623e5b" containerID="c7a6a4518c24e36ee36b7ea1227ec1220716596688b2143524afae7245cc6805" exitCode=22
Nov 03 20:32:39.704371 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:39.704345    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2" event={"ID":"f95421fb-9e1d-4bf8-a721-86323c623e5b","Type":"ContainerDied","Data":"c7a6a4518c24e36ee36b7ea1227ec1220716596688b2143524afae7245cc6805"}
Nov 03 20:32:39.704587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:39.704436    5659 scope.go:122] "RemoveContainer" containerID="b6bb457dd8d431e105f412b891cc5a5f8eea090b2dbb127fd24766dd1742583d"
Nov 03 20:32:39.704661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:39.704646    5659 scope.go:122] "RemoveContainer" containerID="c7a6a4518c24e36ee36b7ea1227ec1220716596688b2143524afae7245cc6805"
Nov 03 20:32:39.705032 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:39.705007    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-vtgk2_odh-ai-gateway-infra(f95421fb-9e1d-4bf8-a721-86323c623e5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2" podUID="f95421fb-9e1d-4bf8-a721-86323c623e5b"
Nov 03 20:32:40.115917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:40.115879    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:40.717394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:40.717364    5659 scope.go:122] "RemoveContainer" containerID="c7a6a4518c24e36ee36b7ea1227ec1220716596688b2143524afae7245cc6805"
Nov 03 20:32:40.717875 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:40.717848    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-vtgk2_odh-ai-gateway-infra(f95421fb-9e1d-4bf8-a721-86323c623e5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2" podUID="f95421fb-9e1d-4bf8-a721-86323c623e5b"
Nov 03 20:32:40.720270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:40.720241    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl" event={"ID":"6b9cc76e-d155-4572-b544-15613ed861e0","Type":"ContainerStarted","Data":"0ce315225d692541d9cd202fe27b64f5231fea6501e9d721745f542816a905a7"}
Nov 03 20:32:40.860577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:40.860543    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:41.665475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:41.665427    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:41.766242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:41.766203    5659 generic.go:356] "Generic (PLEG): container finished" podID="6b9cc76e-d155-4572-b544-15613ed861e0" containerID="0ce315225d692541d9cd202fe27b64f5231fea6501e9d721745f542816a905a7" exitCode=22
Nov 03 20:32:41.767007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:41.766979    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl" event={"ID":"6b9cc76e-d155-4572-b544-15613ed861e0","Type":"ContainerDied","Data":"0ce315225d692541d9cd202fe27b64f5231fea6501e9d721745f542816a905a7"}
Nov 03 20:32:41.767161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:41.767144    5659 scope.go:122] "RemoveContainer" containerID="903983ca48be4d583f6a0295688ebf00cf0e2429d0d9cab95ea88102a372bceb"
Nov 03 20:32:41.767823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:41.767801    5659 scope.go:122] "RemoveContainer" containerID="0ce315225d692541d9cd202fe27b64f5231fea6501e9d721745f542816a905a7"
Nov 03 20:32:41.768454 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:41.768429    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-xtzpl_odh-ai-gateway-infra(6b9cc76e-d155-4572-b544-15613ed861e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl" podUID="6b9cc76e-d155-4572-b544-15613ed861e0"
Nov 03 20:32:42.221757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:42.221723    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:42.364287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:42.364245    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:42.652562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:42.652415    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:42.787503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:42.786528    5659 scope.go:122] "RemoveContainer" containerID="0ce315225d692541d9cd202fe27b64f5231fea6501e9d721745f542816a905a7"
Nov 03 20:32:42.787503 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:42.787006    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-xtzpl_odh-ai-gateway-infra(6b9cc76e-d155-4572-b544-15613ed861e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl" podUID="6b9cc76e-d155-4572-b544-15613ed861e0"
Nov 03 20:32:43.646962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:43.646925    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:44.431708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:44.431662    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:45.189456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:45.189420    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:45.383730 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:45.383691    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:32:45.892975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:45.892937    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:46.699785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:46.699657    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:47.660057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:47.660007    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:48.525778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:48.525728    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:49.334931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:49.334894    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:49.962433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:49.962372    5659 scope.go:122] "RemoveContainer" containerID="c7a6a4518c24e36ee36b7ea1227ec1220716596688b2143524afae7245cc6805"
Nov 03 20:32:50.067521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:50.067392    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:50.686077 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:50.685988    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:50.819228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:50.819189    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:50.905627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:50.905593    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2" event={"ID":"f95421fb-9e1d-4bf8-a721-86323c623e5b","Type":"ContainerStarted","Data":"f190c5d9f1b59b6f617f6fea07951a16b680bc8db8ae6cb4e0d5894586577560"}
Nov 03 20:32:50.956270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:50.956235    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:51.540591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:51.540558    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:51.918018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:51.917938    5659 generic.go:356] "Generic (PLEG): container finished" podID="f95421fb-9e1d-4bf8-a721-86323c623e5b" containerID="f190c5d9f1b59b6f617f6fea07951a16b680bc8db8ae6cb4e0d5894586577560" exitCode=22
Nov 03 20:32:51.918456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:51.918009    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2" event={"ID":"f95421fb-9e1d-4bf8-a721-86323c623e5b","Type":"ContainerDied","Data":"f190c5d9f1b59b6f617f6fea07951a16b680bc8db8ae6cb4e0d5894586577560"}
Nov 03 20:32:51.918456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:51.918079    5659 scope.go:122] "RemoveContainer" containerID="c7a6a4518c24e36ee36b7ea1227ec1220716596688b2143524afae7245cc6805"
Nov 03 20:32:51.918585 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:51.918532    5659 scope.go:122] "RemoveContainer" containerID="f190c5d9f1b59b6f617f6fea07951a16b680bc8db8ae6cb4e0d5894586577560"
Nov 03 20:32:51.919067 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:51.919037    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-266d0609-vtgk2_odh-ai-gateway-infra(f95421fb-9e1d-4bf8-a721-86323c623e5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2" podUID="f95421fb-9e1d-4bf8-a721-86323c623e5b"
Nov 03 20:32:51.942375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:51.942348    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"]
Nov 03 20:32:51.992510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:51.986097    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:52.728903 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:52.728837    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:53.151401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.151371    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"
Nov 03 20:32:53.207432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.207396    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmwn5"]
Nov 03 20:32:53.208688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.208662    5659 cpu_manager.go:414] "containerMap: removing container" podUID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f" containerName="revoke-keys"
Nov 03 20:32:53.208857 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.208841    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f" containerName="revoke-keys"
Nov 03 20:32:53.208975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.208963    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f95421fb-9e1d-4bf8-a721-86323c623e5b" containerName="revoke-keys"
Nov 03 20:32:53.209077 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.209064    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95421fb-9e1d-4bf8-a721-86323c623e5b" containerName="revoke-keys"
Nov 03 20:32:53.209210 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.209197    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f95421fb-9e1d-4bf8-a721-86323c623e5b" containerName="revoke-keys"
Nov 03 20:32:53.209308 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.209295    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95421fb-9e1d-4bf8-a721-86323c623e5b" containerName="revoke-keys"
Nov 03 20:32:53.209456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.209440    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f95421fb-9e1d-4bf8-a721-86323c623e5b" containerName="revoke-keys"
Nov 03 20:32:53.209582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.209568    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95421fb-9e1d-4bf8-a721-86323c623e5b" containerName="revoke-keys"
Nov 03 20:32:53.209897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.209877    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95421fb-9e1d-4bf8-a721-86323c623e5b" containerName="revoke-keys"
Nov 03 20:32:53.210034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.210020    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1342a5-53e9-4494-a78a-d495dfcf4a0f" containerName="revoke-keys"
Nov 03 20:32:53.210137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.210124    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95421fb-9e1d-4bf8-a721-86323c623e5b" containerName="revoke-keys"
Nov 03 20:32:53.216717 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.216689    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmwn5"
Nov 03 20:32:53.221036 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.221010    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 20:32:53.221191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.221103    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vx824\""
Nov 03 20:32:53.227917 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.227396    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmwn5"]
Nov 03 20:32:53.300441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.300341    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95421fb-9e1d-4bf8-a721-86323c623e5b-kube-api-access-gqmrc\" (UniqueName: \"kubernetes.io/projected/f95421fb-9e1d-4bf8-a721-86323c623e5b-kube-api-access-gqmrc\") pod \"f95421fb-9e1d-4bf8-a721-86323c623e5b\" (UID: \"f95421fb-9e1d-4bf8-a721-86323c623e5b\") "
Nov 03 20:32:53.300813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.300566    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-openshift-ca\") pod \"f95421fb-9e1d-4bf8-a721-86323c623e5b\" (UID: \"f95421fb-9e1d-4bf8-a721-86323c623e5b\") "
Nov 03 20:32:53.300813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.300592    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-xks-ca\") pod \"f95421fb-9e1d-4bf8-a721-86323c623e5b\" (UID: \"f95421fb-9e1d-4bf8-a721-86323c623e5b\") "
Nov 03 20:32:53.301314 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.301271    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r6gk\" (UniqueName: \"kubernetes.io/projected/a8606f49-1dc0-45c2-9720-be9731a089c4-kube-api-access-6r6gk\") pod \"dns-operator-controller-manager-8557cbdb84-pmwn5\" (UID: \"a8606f49-1dc0-45c2-9720-be9731a089c4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmwn5"
Nov 03 20:32:53.301545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.301341    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-openshift-ca" pod "f95421fb-9e1d-4bf8-a721-86323c623e5b" (UID: "f95421fb-9e1d-4bf8-a721-86323c623e5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:53.301841 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.301812    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:53.305175 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.303999    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95421fb-9e1d-4bf8-a721-86323c623e5b-kube-api-access-gqmrc" pod "f95421fb-9e1d-4bf8-a721-86323c623e5b" (UID: "f95421fb-9e1d-4bf8-a721-86323c623e5b"). InnerVolumeSpecName "kube-api-access-gqmrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:53.305941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.305916    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-xks-ca" pod "f95421fb-9e1d-4bf8-a721-86323c623e5b" (UID: "f95421fb-9e1d-4bf8-a721-86323c623e5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:53.376035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.375995    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:53.403146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.403107    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6gk\" (UniqueName: \"kubernetes.io/projected/a8606f49-1dc0-45c2-9720-be9731a089c4-kube-api-access-6r6gk\") pod \"dns-operator-controller-manager-8557cbdb84-pmwn5\" (UID: \"a8606f49-1dc0-45c2-9720-be9731a089c4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmwn5"
Nov 03 20:32:53.404470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.404181    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95421fb-9e1d-4bf8-a721-86323c623e5b-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:53.404470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.404211    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqmrc\" (UniqueName: \"kubernetes.io/projected/f95421fb-9e1d-4bf8-a721-86323c623e5b-kube-api-access-gqmrc\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:53.418493 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.418453    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r6gk\" (UniqueName: \"kubernetes.io/projected/a8606f49-1dc0-45c2-9720-be9731a089c4-kube-api-access-6r6gk\") pod \"dns-operator-controller-manager-8557cbdb84-pmwn5\" (UID: \"a8606f49-1dc0-45c2-9720-be9731a089c4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmwn5"
Nov 03 20:32:53.538710 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.538662    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmwn5"
Nov 03 20:32:53.601503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.600793    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:53.930464 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:32:53.930416    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8606f49_1dc0_45c2_9720_be9731a089c4.slice/crio-db16f191284b88318f91fc3462953b817c39727a3b73d6324e548922b321f7fe WatchSource:0}: Error finding container db16f191284b88318f91fc3462953b817c39727a3b73d6324e548922b321f7fe: Status 404 returned error can't find the container with id db16f191284b88318f91fc3462953b817c39727a3b73d6324e548922b321f7fe
Nov 03 20:32:53.931563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.931537    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmwn5"]
Nov 03 20:32:53.944101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.944069    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"
Nov 03 20:32:53.944208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.944105    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2" event={"ID":"f95421fb-9e1d-4bf8-a721-86323c623e5b","Type":"ContainerDied","Data":"28cf7d39b1c8a3eb8d62a30dc1f511b324cf4db2ff9a31394e97bea4664d75c9"}
Nov 03 20:32:53.944208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.944184    5659 scope.go:122] "RemoveContainer" containerID="f190c5d9f1b59b6f617f6fea07951a16b680bc8db8ae6cb4e0d5894586577560"
Nov 03 20:32:53.962134 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.962111    5659 scope.go:122] "RemoveContainer" containerID="0ce315225d692541d9cd202fe27b64f5231fea6501e9d721745f542816a905a7"
Nov 03 20:32:53.987570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.987533    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"]
Nov 03 20:32:53.992349 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:53.992324    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-266d0609-vtgk2"]
Nov 03 20:32:54.496895 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:54.481026    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:54.958028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:54.957979    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl" event={"ID":"6b9cc76e-d155-4572-b544-15613ed861e0","Type":"ContainerStarted","Data":"0304dea0c7ef67e9931200afd445cbeab5fdd207610eb4a764a5afbf47f12be7"}
Nov 03 20:32:54.959694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:54.959663    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmwn5" event={"ID":"a8606f49-1dc0-45c2-9720-be9731a089c4","Type":"ContainerStarted","Data":"db16f191284b88318f91fc3462953b817c39727a3b73d6324e548922b321f7fe"}
Nov 03 20:32:55.338577 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:55.337876    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:55.974304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:55.974262    5659 generic.go:356] "Generic (PLEG): container finished" podID="6b9cc76e-d155-4572-b544-15613ed861e0" containerID="0304dea0c7ef67e9931200afd445cbeab5fdd207610eb4a764a5afbf47f12be7" exitCode=22
Nov 03 20:32:55.978064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:55.978038    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95421fb-9e1d-4bf8-a721-86323c623e5b" path="/var/lib/kubelet/pods/f95421fb-9e1d-4bf8-a721-86323c623e5b/volumes"
Nov 03 20:32:55.979037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:55.979005    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl" event={"ID":"6b9cc76e-d155-4572-b544-15613ed861e0","Type":"ContainerDied","Data":"0304dea0c7ef67e9931200afd445cbeab5fdd207610eb4a764a5afbf47f12be7"}
Nov 03 20:32:55.979150 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:55.979088    5659 scope.go:122] "RemoveContainer" containerID="0ce315225d692541d9cd202fe27b64f5231fea6501e9d721745f542816a905a7"
Nov 03 20:32:55.979698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:55.979675    5659 scope.go:122] "RemoveContainer" containerID="0304dea0c7ef67e9931200afd445cbeab5fdd207610eb4a764a5afbf47f12be7"
Nov 03 20:32:55.982763 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:32:55.982734    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-5bac6f1e-xtzpl_odh-ai-gateway-infra(6b9cc76e-d155-4572-b544-15613ed861e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl" podUID="6b9cc76e-d155-4572-b544-15613ed861e0"
Nov 03 20:32:55.995543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:55.994813    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"]
Nov 03 20:32:56.393962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:56.393856    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:57.352978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.352906    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:57.526677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.526646    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:57.566051 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.566022    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"
Nov 03 20:32:57.704188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.704155    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-openshift-ca\") pod \"6b9cc76e-d155-4572-b544-15613ed861e0\" (UID: \"6b9cc76e-d155-4572-b544-15613ed861e0\") "
Nov 03 20:32:57.704376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.704326    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9cc76e-d155-4572-b544-15613ed861e0-kube-api-access-wcdbq\" (UniqueName: \"kubernetes.io/projected/6b9cc76e-d155-4572-b544-15613ed861e0-kube-api-access-wcdbq\") pod \"6b9cc76e-d155-4572-b544-15613ed861e0\" (UID: \"6b9cc76e-d155-4572-b544-15613ed861e0\") "
Nov 03 20:32:57.704450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.704440    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-xks-ca\") pod \"6b9cc76e-d155-4572-b544-15613ed861e0\" (UID: \"6b9cc76e-d155-4572-b544-15613ed861e0\") "
Nov 03 20:32:57.704548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.704520    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-openshift-ca" pod "6b9cc76e-d155-4572-b544-15613ed861e0" (UID: "6b9cc76e-d155-4572-b544-15613ed861e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:57.705014 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.704994    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:57.707691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.707667    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-xks-ca" pod "6b9cc76e-d155-4572-b544-15613ed861e0" (UID: "6b9cc76e-d155-4572-b544-15613ed861e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:57.707796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.707744    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9cc76e-d155-4572-b544-15613ed861e0-kube-api-access-wcdbq" pod "6b9cc76e-d155-4572-b544-15613ed861e0" (UID: "6b9cc76e-d155-4572-b544-15613ed861e0"). InnerVolumeSpecName "kube-api-access-wcdbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:57.722065 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.722025    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"]
Nov 03 20:32:57.722877 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.722854    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6b9cc76e-d155-4572-b544-15613ed861e0" containerName="revoke-keys"
Nov 03 20:32:57.722877 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.722876    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9cc76e-d155-4572-b544-15613ed861e0" containerName="revoke-keys"
Nov 03 20:32:57.723086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.722911    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6b9cc76e-d155-4572-b544-15613ed861e0" containerName="revoke-keys"
Nov 03 20:32:57.723086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.722917    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9cc76e-d155-4572-b544-15613ed861e0" containerName="revoke-keys"
Nov 03 20:32:57.723207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.723147    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95421fb-9e1d-4bf8-a721-86323c623e5b" containerName="revoke-keys"
Nov 03 20:32:57.723207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.723166    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9cc76e-d155-4572-b544-15613ed861e0" containerName="revoke-keys"
Nov 03 20:32:57.723207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.723187    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9cc76e-d155-4572-b544-15613ed861e0" containerName="revoke-keys"
Nov 03 20:32:57.727305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.727281    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"
Nov 03 20:32:57.741393 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.741345    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"]
Nov 03 20:32:57.814787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.809217    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-lblgz\" (UID: \"b9851d73-2717-41bd-8043-fb0557d4aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"
Nov 03 20:32:57.814787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.809439    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pffn2\" (UniqueName: \"kubernetes.io/projected/b9851d73-2717-41bd-8043-fb0557d4aa93-kube-api-access-pffn2\") pod \"maas-api-revoke-sub-f14db2bd-lblgz\" (UID: \"b9851d73-2717-41bd-8043-fb0557d4aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"
Nov 03 20:32:57.814787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.809590    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-lblgz\" (UID: \"b9851d73-2717-41bd-8043-fb0557d4aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"
Nov 03 20:32:57.814787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.809788    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9cc76e-d155-4572-b544-15613ed861e0-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:57.814787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.809806    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcdbq\" (UniqueName: \"kubernetes.io/projected/6b9cc76e-d155-4572-b544-15613ed861e0-kube-api-access-wcdbq\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:57.914235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.913852    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pffn2\" (UniqueName: \"kubernetes.io/projected/b9851d73-2717-41bd-8043-fb0557d4aa93-kube-api-access-pffn2\") pod \"maas-api-revoke-sub-f14db2bd-lblgz\" (UID: \"b9851d73-2717-41bd-8043-fb0557d4aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"
Nov 03 20:32:57.914235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.914026    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-lblgz\" (UID: \"b9851d73-2717-41bd-8043-fb0557d4aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"
Nov 03 20:32:57.914235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.914161    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-lblgz\" (UID: \"b9851d73-2717-41bd-8043-fb0557d4aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"
Nov 03 20:32:57.916268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.915284    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14db2bd-lblgz\" (UID: \"b9851d73-2717-41bd-8043-fb0557d4aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"
Nov 03 20:32:57.921266 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.921164    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14db2bd-lblgz\" (UID: \"b9851d73-2717-41bd-8043-fb0557d4aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"
Nov 03 20:32:57.927452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:57.927425    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pffn2\" (UniqueName: \"kubernetes.io/projected/b9851d73-2717-41bd-8043-fb0557d4aa93-kube-api-access-pffn2\") pod \"maas-api-revoke-sub-f14db2bd-lblgz\" (UID: \"b9851d73-2717-41bd-8043-fb0557d4aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"
Nov 03 20:32:58.006136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:58.006109    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"
Nov 03 20:32:58.006293 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:58.006129    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl" event={"ID":"6b9cc76e-d155-4572-b544-15613ed861e0","Type":"ContainerDied","Data":"22af37dd16edfb3c46f4175c6a49e149e3b3055cc31a538b9a6bbdfe9d7168b2"}
Nov 03 20:32:58.006293 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:58.006200    5659 scope.go:122] "RemoveContainer" containerID="0304dea0c7ef67e9931200afd445cbeab5fdd207610eb4a764a5afbf47f12be7"
Nov 03 20:32:58.009636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:58.009609    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmwn5" event={"ID":"a8606f49-1dc0-45c2-9720-be9731a089c4","Type":"ContainerStarted","Data":"878a39b80767d2e0c7b40de7462571b6b3a8076465b7878b50ea7efd7d2633f7"}
Nov 03 20:32:58.009834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:58.009814    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmwn5"
Nov 03 20:32:58.055557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:58.055071    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"
Nov 03 20:32:58.108517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:58.105576    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmwn5" podStartSLOduration=51.24277105 podStartE2EDuration="55.105555101s" totalImagesPullingTime="3.862784051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:58.049305404 +0000 UTC m=+3330.204207825" watchObservedRunningTime="2026-11-03 20:32:58.105555101 +0000 UTC m=+3330.260457520"
Nov 03 20:32:58.118154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:58.116959    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"]
Nov 03 20:32:58.134515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:58.133502    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac6f1e-xtzpl"]
Nov 03 20:32:58.316560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:58.310046    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:58.874726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:58.874691    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"]
Nov 03 20:32:59.076114 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:59.074583    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:59.144513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:59.118386    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz" event={"ID":"b9851d73-2717-41bd-8043-fb0557d4aa93","Type":"ContainerStarted","Data":"e06a350561894c12cc0744eb2af6e4003d62d2f5219596003fcdcca89c61fbdf"}
Nov 03 20:32:59.947175 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:32:59.947077    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:00.053502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:00.053297    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9cc76e-d155-4572-b544-15613ed861e0" path="/var/lib/kubelet/pods/6b9cc76e-d155-4572-b544-15613ed861e0/volumes"
Nov 03 20:33:00.228647 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:00.228612    5659 generic.go:356] "Generic (PLEG): container finished" podID="b9851d73-2717-41bd-8043-fb0557d4aa93" containerID="bbf9716be48619836a0743e4a37e6041c389fbefad8e639c799919b3df13e6d0" exitCode=7
Nov 03 20:33:00.228844 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:00.228703    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz" event={"ID":"b9851d73-2717-41bd-8043-fb0557d4aa93","Type":"ContainerDied","Data":"bbf9716be48619836a0743e4a37e6041c389fbefad8e639c799919b3df13e6d0"}
Nov 03 20:33:00.229090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:00.229074    5659 scope.go:122] "RemoveContainer" containerID="bbf9716be48619836a0743e4a37e6041c389fbefad8e639c799919b3df13e6d0"
Nov 03 20:33:01.074135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:01.074098    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:01.300559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:01.300386    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz" event={"ID":"b9851d73-2717-41bd-8043-fb0557d4aa93","Type":"ContainerStarted","Data":"bbc1988c6631683b0c2867413c1eeeb9f0839c47ff059c9654263afe614a8faf"}
Nov 03 20:33:01.324132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:01.324077    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz" podStartSLOduration=41.324062826 podStartE2EDuration="41.324062826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:01.323231617 +0000 UTC m=+3333.478134035" watchObservedRunningTime="2026-11-03 20:33:01.324062826 +0000 UTC m=+3333.478965271"
Nov 03 20:33:01.956017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:01.955979    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:02.324612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:02.324513    5659 generic.go:356] "Generic (PLEG): container finished" podID="b9851d73-2717-41bd-8043-fb0557d4aa93" containerID="bbc1988c6631683b0c2867413c1eeeb9f0839c47ff059c9654263afe614a8faf" exitCode=7
Nov 03 20:33:02.324612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:02.324564    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz" event={"ID":"b9851d73-2717-41bd-8043-fb0557d4aa93","Type":"ContainerDied","Data":"bbc1988c6631683b0c2867413c1eeeb9f0839c47ff059c9654263afe614a8faf"}
Nov 03 20:33:02.325143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:02.324641    5659 scope.go:122] "RemoveContainer" containerID="bbf9716be48619836a0743e4a37e6041c389fbefad8e639c799919b3df13e6d0"
Nov 03 20:33:02.325143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:02.324974    5659 scope.go:122] "RemoveContainer" containerID="bbc1988c6631683b0c2867413c1eeeb9f0839c47ff059c9654263afe614a8faf"
Nov 03 20:33:02.325444 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:33:02.325421    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-lblgz_odh-ai-gateway-infra(b9851d73-2717-41bd-8043-fb0557d4aa93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz" podUID="b9851d73-2717-41bd-8043-fb0557d4aa93"
Nov 03 20:33:02.859876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:02.859838    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:03.009376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:03.009340    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:03.337242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:03.337213    5659 scope.go:122] "RemoveContainer" containerID="bbc1988c6631683b0c2867413c1eeeb9f0839c47ff059c9654263afe614a8faf"
Nov 03 20:33:03.337762 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:33:03.337732    5659 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-f14db2bd-lblgz_odh-ai-gateway-infra(b9851d73-2717-41bd-8043-fb0557d4aa93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz" podUID="b9851d73-2717-41bd-8043-fb0557d4aa93"
Nov 03 20:33:03.829756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:03.829724    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:04.644635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:04.644597    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:05.544290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:05.544252    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:05.692505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:05.685184    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:06.083405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:06.083370    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:06.414794 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:06.414467    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:06.583837 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:06.583793    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:07.442604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:07.442445    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:08.263975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:08.263879    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:09.154921 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:09.154833    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:09.261131 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:09.250196    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pmwn5"
Nov 03 20:33:10.190324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:10.190232    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:11.164328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:11.164143    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:11.988011 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:11.987976    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:12.860770 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:12.860718    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:13.671535 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:13.669680    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:13.974844 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:13.974814    5659 scope.go:122] "RemoveContainer" containerID="bbc1988c6631683b0c2867413c1eeeb9f0839c47ff059c9654263afe614a8faf"
Nov 03 20:33:14.520928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:14.520826    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz" event={"ID":"b9851d73-2717-41bd-8043-fb0557d4aa93","Type":"ContainerStarted","Data":"04da7507736690f268e14e27e7758603ea25b5493476514ec3c32f7dd3d7d572"}
Nov 03 20:33:14.540619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:14.540412    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:15.373808 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:33:15.357154    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:33:15.606888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:15.606795    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"]
Nov 03 20:33:15.610010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:15.609425    5659 generic.go:356] "Generic (PLEG): container finished" podID="b9851d73-2717-41bd-8043-fb0557d4aa93" containerID="04da7507736690f268e14e27e7758603ea25b5493476514ec3c32f7dd3d7d572" exitCode=7
Nov 03 20:33:15.610010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:15.609477    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz" event={"ID":"b9851d73-2717-41bd-8043-fb0557d4aa93","Type":"ContainerDied","Data":"04da7507736690f268e14e27e7758603ea25b5493476514ec3c32f7dd3d7d572"}
Nov 03 20:33:15.610010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:15.609537    5659 scope.go:122] "RemoveContainer" containerID="bbc1988c6631683b0c2867413c1eeeb9f0839c47ff059c9654263afe614a8faf"
Nov 03 20:33:16.175504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:15.992981    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnfsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:33:16.175504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.171746    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" podUID="9fa9e0c2-1bfa-4b41-b033-8d286b4c63be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:33:16.199700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.199663    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:16.307735 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.307683    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"
Nov 03 20:33:16.428380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.428278    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-xks-ca\") pod \"b9851d73-2717-41bd-8043-fb0557d4aa93\" (UID: \"b9851d73-2717-41bd-8043-fb0557d4aa93\") "
Nov 03 20:33:16.428929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.428534    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9851d73-2717-41bd-8043-fb0557d4aa93-kube-api-access-pffn2\" (UniqueName: \"kubernetes.io/projected/b9851d73-2717-41bd-8043-fb0557d4aa93-kube-api-access-pffn2\") pod \"b9851d73-2717-41bd-8043-fb0557d4aa93\" (UID: \"b9851d73-2717-41bd-8043-fb0557d4aa93\") "
Nov 03 20:33:16.428929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.428680    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-openshift-ca\") pod \"b9851d73-2717-41bd-8043-fb0557d4aa93\" (UID: \"b9851d73-2717-41bd-8043-fb0557d4aa93\") "
Nov 03 20:33:16.429101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.429069    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-openshift-ca" pod "b9851d73-2717-41bd-8043-fb0557d4aa93" (UID: "b9851d73-2717-41bd-8043-fb0557d4aa93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:16.429678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.429653    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:16.430781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.430756    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-xks-ca" pod "b9851d73-2717-41bd-8043-fb0557d4aa93" (UID: "b9851d73-2717-41bd-8043-fb0557d4aa93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:16.431427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.431409    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9851d73-2717-41bd-8043-fb0557d4aa93-kube-api-access-pffn2" pod "b9851d73-2717-41bd-8043-fb0557d4aa93" (UID: "b9851d73-2717-41bd-8043-fb0557d4aa93"). InnerVolumeSpecName "kube-api-access-pffn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:16.531221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.531187    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9851d73-2717-41bd-8043-fb0557d4aa93-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:16.531221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.531220    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pffn2\" (UniqueName: \"kubernetes.io/projected/b9851d73-2717-41bd-8043-fb0557d4aa93-kube-api-access-pffn2\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:16.625423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.625390    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"
Nov 03 20:33:16.625669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.625446    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz" event={"ID":"b9851d73-2717-41bd-8043-fb0557d4aa93","Type":"ContainerDied","Data":"e06a350561894c12cc0744eb2af6e4003d62d2f5219596003fcdcca89c61fbdf"}
Nov 03 20:33:16.625669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.625558    5659 scope.go:122] "RemoveContainer" containerID="04da7507736690f268e14e27e7758603ea25b5493476514ec3c32f7dd3d7d572"
Nov 03 20:33:16.667987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.667951    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"]
Nov 03 20:33:16.673222 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.673183    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14db2bd-lblgz"]
Nov 03 20:33:16.940505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:16.939654    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:17.493682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:17.493646    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:17.623343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:17.623021    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:17.977851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:17.977811    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9851d73-2717-41bd-8043-fb0557d4aa93" path="/var/lib/kubelet/pods/b9851d73-2717-41bd-8043-fb0557d4aa93/volumes"
Nov 03 20:33:18.273472 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:18.273298    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:19.061228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:19.061161    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:19.737115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:19.737074    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:20.419945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:20.419907    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:21.201396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:21.201357    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:21.609451 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:21.609414    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:21.829723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:21.829690    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:22.756345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:22.752860    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:23.545680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:23.545638    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:24.325761 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:24.325727    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:25.021182 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:25.021149    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:25.185978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:25.178187    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:25.958016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:25.957978    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:26.689505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:26.688533    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:27.463513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:27.451127    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:28.485505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:28.484701    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:29.336035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:29.335998    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:29.850553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:29.850506    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"]
Nov 03 20:33:29.851085 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:29.850752    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" podUID="cc874822-913a-4d2d-922f-19d6095c34f9" containerName="istio-proxy" containerID="cri-o://393c2290a69b81efe255c23e740d2763fbc15702f66320dbb1cf9a90d6a5f860" gracePeriod=30
Nov 03 20:33:29.857967 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:29.857933    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"]
Nov 03 20:33:29.858208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:29.858181    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" podUID="15cce432-d79b-4f87-b5c6-63908192552d" containerName="istio-proxy" containerID="cri-o://3c2ea4341de31f162e7ba96742d68a316e58807241967c5b3bda4b3c7d480044" gracePeriod=30
Nov 03 20:33:30.075301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:30.075237    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:31.047190 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:31.047154    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:31.809634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:31.809598    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:32.545206 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:32.545111    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:33.442640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:33.442603    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:34.203348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:34.203308    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:34.863942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:34.863904    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:35.187393 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.187345    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:33:35.214735 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.214701    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-socket\") pod \"cc874822-913a-4d2d-922f-19d6095c34f9\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") "
Nov 03 20:33:35.215262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.214768    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-istio-token\" (UniqueName: \"kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-istio-token\") pod \"cc874822-913a-4d2d-922f-19d6095c34f9\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") "
Nov 03 20:33:35.215262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.214844    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc874822-913a-4d2d-922f-19d6095c34f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc874822-913a-4d2d-922f-19d6095c34f9-istio-podinfo\") pod \"cc874822-913a-4d2d-922f-19d6095c34f9\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") "
Nov 03 20:33:35.215262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.215008    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-socket" pod "cc874822-913a-4d2d-922f-19d6095c34f9" (UID: "cc874822-913a-4d2d-922f-19d6095c34f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:35.215826 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.215795    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-kube-api-access-zzghp\" (UniqueName: \"kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-kube-api-access-zzghp\") pod \"cc874822-913a-4d2d-922f-19d6095c34f9\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") "
Nov 03 20:33:35.216000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.215883    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc874822-913a-4d2d-922f-19d6095c34f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc874822-913a-4d2d-922f-19d6095c34f9-istiod-ca-cert\") pod \"cc874822-913a-4d2d-922f-19d6095c34f9\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") "
Nov 03 20:33:35.216000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.215933    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-credential-socket\") pod \"cc874822-913a-4d2d-922f-19d6095c34f9\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") "
Nov 03 20:33:35.216000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.215982    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-envoy\") pod \"cc874822-913a-4d2d-922f-19d6095c34f9\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") "
Nov 03 20:33:35.216747 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.216111    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-data\") pod \"cc874822-913a-4d2d-922f-19d6095c34f9\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") "
Nov 03 20:33:35.216747 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.216219    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-certs\") pod \"cc874822-913a-4d2d-922f-19d6095c34f9\" (UID: \"cc874822-913a-4d2d-922f-19d6095c34f9\") "
Nov 03 20:33:35.217172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.216948    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc874822-913a-4d2d-922f-19d6095c34f9-istiod-ca-cert" pod "cc874822-913a-4d2d-922f-19d6095c34f9" (UID: "cc874822-913a-4d2d-922f-19d6095c34f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:35.217172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.217154    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-data" pod "cc874822-913a-4d2d-922f-19d6095c34f9" (UID: "cc874822-913a-4d2d-922f-19d6095c34f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:35.217346 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.217166    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-certs" pod "cc874822-913a-4d2d-922f-19d6095c34f9" (UID: "cc874822-913a-4d2d-922f-19d6095c34f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:35.217496 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.217453    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-credential-socket" pod "cc874822-913a-4d2d-922f-19d6095c34f9" (UID: "cc874822-913a-4d2d-922f-19d6095c34f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:35.217899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.217875    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.218320 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.217923    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.218320 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.217947    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc874822-913a-4d2d-922f-19d6095c34f9-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.218320 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.217961    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.218320 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.217974    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.219887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.219862    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-envoy" pod "cc874822-913a-4d2d-922f-19d6095c34f9" (UID: "cc874822-913a-4d2d-922f-19d6095c34f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:35.220139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.220114    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc874822-913a-4d2d-922f-19d6095c34f9-istio-podinfo" pod "cc874822-913a-4d2d-922f-19d6095c34f9" (UID: "cc874822-913a-4d2d-922f-19d6095c34f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:33:35.220304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.220284    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-kube-api-access-zzghp" pod "cc874822-913a-4d2d-922f-19d6095c34f9" (UID: "cc874822-913a-4d2d-922f-19d6095c34f9"). InnerVolumeSpecName "kube-api-access-zzghp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:35.221200 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.221176    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-istio-token" pod "cc874822-913a-4d2d-922f-19d6095c34f9" (UID: "cc874822-913a-4d2d-922f-19d6095c34f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:35.251912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.251874    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:33:35.319786 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.319742    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-envoy\") pod \"15cce432-d79b-4f87-b5c6-63908192552d\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") "
Nov 03 20:33:35.319990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.319816    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-data\") pod \"15cce432-d79b-4f87-b5c6-63908192552d\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") "
Nov 03 20:33:35.319990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.319914    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-certs\") pod \"15cce432-d79b-4f87-b5c6-63908192552d\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") "
Nov 03 20:33:35.319990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.319973    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-credential-socket\") pod \"15cce432-d79b-4f87-b5c6-63908192552d\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") "
Nov 03 20:33:35.320192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.320035    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-kube-api-access-gwkc2\" (UniqueName: \"kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-kube-api-access-gwkc2\") pod \"15cce432-d79b-4f87-b5c6-63908192552d\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") "
Nov 03 20:33:35.320192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.320104    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15cce432-d79b-4f87-b5c6-63908192552d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15cce432-d79b-4f87-b5c6-63908192552d-istio-podinfo\") pod \"15cce432-d79b-4f87-b5c6-63908192552d\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") "
Nov 03 20:33:35.320192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.320138    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-data" pod "15cce432-d79b-4f87-b5c6-63908192552d" (UID: "15cce432-d79b-4f87-b5c6-63908192552d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:35.320192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.320180    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15cce432-d79b-4f87-b5c6-63908192552d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15cce432-d79b-4f87-b5c6-63908192552d-istiod-ca-cert\") pod \"15cce432-d79b-4f87-b5c6-63908192552d\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") "
Nov 03 20:33:35.320427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.320216    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-socket\") pod \"15cce432-d79b-4f87-b5c6-63908192552d\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") "
Nov 03 20:33:35.320427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.320233    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-certs" pod "15cce432-d79b-4f87-b5c6-63908192552d" (UID: "15cce432-d79b-4f87-b5c6-63908192552d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:35.320427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.320260    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-istio-token\" (UniqueName: \"kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-istio-token\") pod \"15cce432-d79b-4f87-b5c6-63908192552d\" (UID: \"15cce432-d79b-4f87-b5c6-63908192552d\") "
Nov 03 20:33:35.320427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.320266    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-credential-socket" pod "15cce432-d79b-4f87-b5c6-63908192552d" (UID: "15cce432-d79b-4f87-b5c6-63908192552d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:35.320766 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.320561    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15cce432-d79b-4f87-b5c6-63908192552d-istiod-ca-cert" pod "15cce432-d79b-4f87-b5c6-63908192552d" (UID: "15cce432-d79b-4f87-b5c6-63908192552d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:35.320924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.320898    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-socket" pod "15cce432-d79b-4f87-b5c6-63908192552d" (UID: "15cce432-d79b-4f87-b5c6-63908192552d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:35.321117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.321095    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzghp\" (UniqueName: \"kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-kube-api-access-zzghp\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.321193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.321124    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc874822-913a-4d2d-922f-19d6095c34f9-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.321193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.321139    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.321193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.321154    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.321193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.321171    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.321193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.321184    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15cce432-d79b-4f87-b5c6-63908192552d-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.321500 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.321199    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.321500 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.321216    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc874822-913a-4d2d-922f-19d6095c34f9-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.321500 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.321230    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc874822-913a-4d2d-922f-19d6095c34f9-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.322769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.322741    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-envoy" pod "15cce432-d79b-4f87-b5c6-63908192552d" (UID: "15cce432-d79b-4f87-b5c6-63908192552d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:35.323648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.323402    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-kube-api-access-gwkc2" pod "15cce432-d79b-4f87-b5c6-63908192552d" (UID: "15cce432-d79b-4f87-b5c6-63908192552d"). InnerVolumeSpecName "kube-api-access-gwkc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:35.323648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.323436    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-istio-token" pod "15cce432-d79b-4f87-b5c6-63908192552d" (UID: "15cce432-d79b-4f87-b5c6-63908192552d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:35.323991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.323966    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15cce432-d79b-4f87-b5c6-63908192552d-istio-podinfo" pod "15cce432-d79b-4f87-b5c6-63908192552d" (UID: "15cce432-d79b-4f87-b5c6-63908192552d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:33:35.423699 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.423599    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwkc2\" (UniqueName: \"kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-kube-api-access-gwkc2\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.423699 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.423637    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15cce432-d79b-4f87-b5c6-63908192552d-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.423699 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.423654    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15cce432-d79b-4f87-b5c6-63908192552d-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.423699 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.423667    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15cce432-d79b-4f87-b5c6-63908192552d-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:35.632548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.632518    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:35.873110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.873069    5659 generic.go:356] "Generic (PLEG): container finished" podID="15cce432-d79b-4f87-b5c6-63908192552d" containerID="3c2ea4341de31f162e7ba96742d68a316e58807241967c5b3bda4b3c7d480044" exitCode=0
Nov 03 20:33:35.873376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.873125    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"
Nov 03 20:33:35.873376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.873144    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" event={"ID":"15cce432-d79b-4f87-b5c6-63908192552d","Type":"ContainerDied","Data":"3c2ea4341de31f162e7ba96742d68a316e58807241967c5b3bda4b3c7d480044"}
Nov 03 20:33:35.873376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.873209    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97" event={"ID":"15cce432-d79b-4f87-b5c6-63908192552d","Type":"ContainerDied","Data":"fa21a220c37bc8e9e7466e33ca54a91303c5d26d9222d9401e324b4272678384"}
Nov 03 20:33:35.873376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.873238    5659 scope.go:122] "RemoveContainer" containerID="3c2ea4341de31f162e7ba96742d68a316e58807241967c5b3bda4b3c7d480044"
Nov 03 20:33:35.874879 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.874853    5659 generic.go:356] "Generic (PLEG): container finished" podID="cc874822-913a-4d2d-922f-19d6095c34f9" containerID="393c2290a69b81efe255c23e740d2763fbc15702f66320dbb1cf9a90d6a5f860" exitCode=0
Nov 03 20:33:35.874988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.874897    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"
Nov 03 20:33:35.875127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.875109    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" event={"ID":"cc874822-913a-4d2d-922f-19d6095c34f9","Type":"ContainerDied","Data":"393c2290a69b81efe255c23e740d2763fbc15702f66320dbb1cf9a90d6a5f860"}
Nov 03 20:33:35.875202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.875154    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q" event={"ID":"cc874822-913a-4d2d-922f-19d6095c34f9","Type":"ContainerDied","Data":"71828ff694fba8f6a54253e82abcc0c8ad7dbc7a6ccbb112a1048c5a590cfaaa"}
Nov 03 20:33:35.915193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.915160    5659 scope.go:122] "RemoveContainer" containerID="3c2ea4341de31f162e7ba96742d68a316e58807241967c5b3bda4b3c7d480044"
Nov 03 20:33:35.915907 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:33:35.915874    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2ea4341de31f162e7ba96742d68a316e58807241967c5b3bda4b3c7d480044\": container with ID starting with 3c2ea4341de31f162e7ba96742d68a316e58807241967c5b3bda4b3c7d480044 not found: ID does not exist" containerID="3c2ea4341de31f162e7ba96742d68a316e58807241967c5b3bda4b3c7d480044"
Nov 03 20:33:35.916055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.915912    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2ea4341de31f162e7ba96742d68a316e58807241967c5b3bda4b3c7d480044"} err="failed to get container status \"3c2ea4341de31f162e7ba96742d68a316e58807241967c5b3bda4b3c7d480044\": rpc error: code = NotFound desc = could not find container \"3c2ea4341de31f162e7ba96742d68a316e58807241967c5b3bda4b3c7d480044\": container with ID starting with 3c2ea4341de31f162e7ba96742d68a316e58807241967c5b3bda4b3c7d480044 not found: ID does not exist"
Nov 03 20:33:35.916055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.915937    5659 scope.go:122] "RemoveContainer" containerID="393c2290a69b81efe255c23e740d2763fbc15702f66320dbb1cf9a90d6a5f860"
Nov 03 20:33:35.922155 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.922126    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"]
Nov 03 20:33:35.926863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.926839    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-pfb97"]
Nov 03 20:33:35.931531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.931449    5659 scope.go:122] "RemoveContainer" containerID="393c2290a69b81efe255c23e740d2763fbc15702f66320dbb1cf9a90d6a5f860"
Nov 03 20:33:35.931812 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:33:35.931792    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393c2290a69b81efe255c23e740d2763fbc15702f66320dbb1cf9a90d6a5f860\": container with ID starting with 393c2290a69b81efe255c23e740d2763fbc15702f66320dbb1cf9a90d6a5f860 not found: ID does not exist" containerID="393c2290a69b81efe255c23e740d2763fbc15702f66320dbb1cf9a90d6a5f860"
Nov 03 20:33:35.931896 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.931823    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393c2290a69b81efe255c23e740d2763fbc15702f66320dbb1cf9a90d6a5f860"} err="failed to get container status \"393c2290a69b81efe255c23e740d2763fbc15702f66320dbb1cf9a90d6a5f860\": rpc error: code = NotFound desc = could not find container \"393c2290a69b81efe255c23e740d2763fbc15702f66320dbb1cf9a90d6a5f860\": container with ID starting with 393c2290a69b81efe255c23e740d2763fbc15702f66320dbb1cf9a90d6a5f860 not found: ID does not exist"
Nov 03 20:33:35.936137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.936108    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"]
Nov 03 20:33:35.940380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.940354    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-w845q"]
Nov 03 20:33:35.973149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.973115    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15cce432-d79b-4f87-b5c6-63908192552d" path="/var/lib/kubelet/pods/15cce432-d79b-4f87-b5c6-63908192552d/volumes"
Nov 03 20:33:35.973901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:35.973875    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc874822-913a-4d2d-922f-19d6095c34f9" path="/var/lib/kubelet/pods/cc874822-913a-4d2d-922f-19d6095c34f9/volumes"
Nov 03 20:33:36.535544 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:36.534044    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:37.188624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:37.188584    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:38.019522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:38.011017    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:38.858510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:38.849991    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:39.644556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:39.643323    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:40.453567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:40.453514    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:41.135053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:41.135007    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:41.923670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:41.923635    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:42.745794 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:42.745746    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:43.575505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:43.565026    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:44.262035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:44.261992    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:45.162506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:45.159286    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:45.250343 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:33:45.250303    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:33:45.805500 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:45.805455    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:46.582502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:46.581693    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:47.404534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:47.401466    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:48.068510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:48.068317    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:48.939097 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:48.939052    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:49.653753 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:49.653717    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:50.387922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:50.387829    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:51.110934 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:51.110897    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:51.995975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:51.995936    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:52.729720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:52.729687    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:53.522652 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:53.522540    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:54.295631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:54.295596    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:54.993299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:54.993224    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:55.836750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:55.836709    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:56.822132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:56.822094    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:57.545958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:57.545907    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:58.231868 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:58.231829    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:59.058326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:59.058284    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:59.923489 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:33:59.923440    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:00.735306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:00.735263    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:01.447757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:01.447660    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:02.140509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:02.139966    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:02.802920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:02.802876    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:03.621411 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:03.621368    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:04.543392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:04.543342    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:05.327367 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:05.327327    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:06.144198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:06.144154    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:06.287713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:06.287674    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:06.910267 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:06.910230    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:07.739294 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:07.739246    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:08.448873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:08.448833    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:09.172704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:09.172612    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:09.932052 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:09.932008    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:10.754507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:10.753965    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:11.313672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:11.313640    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:11.612681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:11.612595    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:12.491507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:12.487192    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:13.284829 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:13.284788    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:13.995035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:13.994993    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:14.897514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:14.893153    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:15.220871 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:34:15.220834    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:34:15.648994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:15.648906    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:16.413545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:16.401915    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:17.275051 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:17.275014    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:18.116631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:18.116589    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:18.942423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:18.942380    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:19.735852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:19.735817    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:20.437955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:20.437914    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:21.292089 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:21.292057    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:22.132656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:22.132624    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:22.834094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:22.834024    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:23.631579 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:23.631539    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:23.964623 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:23.964587    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:24.111756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:24.111663    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:24.240834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:24.240749    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:24.522963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:24.522873    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:25.226508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:25.218997    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:25.711736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:25.711704    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:25.849060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:25.849024    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:26.049964 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:26.049879    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:26.633164 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:26.633128    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:27.419198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:27.419160    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:27.675313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:27.675213    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:28.155528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:28.155464    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:28.916947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:28.916913    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:29.083923 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:29.083880    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:29.582334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:29.582301    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:30.255431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:30.255400    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:30.924362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:30.924327    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:31.241856 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:31.241816    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:31.375933 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:31.375893    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:31.608455 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:31.608364    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:31.747854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:31.744990    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:32.271218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:32.271179    5659 scope.go:122] "RemoveContainer" containerID="0932c977f449700e2547ce96a785cf5ecd868a615d9a0157d839cb760c780d73"
Nov 03 20:34:32.537591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:32.535319    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:33.219154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:33.219121    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:33.864798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:33.864761    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:34.622681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:34.622639    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:35.446726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:35.446673    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:36.171254 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:36.171217    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:36.929845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:36.929803    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:37.732309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:37.732249    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:38.192782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:38.192693    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"]
Nov 03 20:34:38.293746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:38.293705    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:38.439004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:38.438966    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:38.968560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:38.963060    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:39.398223 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:39.398139    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:39.528953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:39.528917    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:39.854139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:39.854104    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:40.435942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:40.435907    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:40.642505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:40.638815    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:41.517099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:41.517036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:42.245398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:42.245339    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:43.047912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.047868    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:43.193405 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.193365    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7" podUID="de7c4c4d-2e02-4c99-b9e0-8f15391d7871" containerName="payload-pre-processing" containerID="cri-o://e371e3e73b5059592ffdc3fbc899049cb319e3e96b864bba71471a04e5931da7" gracePeriod=25
Nov 03 20:34:43.544929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.544892    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"
Nov 03 20:34:43.633150 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.633051    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-kube-api-access-bb86n\" (UniqueName: \"kubernetes.io/projected/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-kube-api-access-bb86n\") pod \"de7c4c4d-2e02-4c99-b9e0-8f15391d7871\" (UID: \"de7c4c4d-2e02-4c99-b9e0-8f15391d7871\") "
Nov 03 20:34:43.633150 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.633115    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-plugins-config-volume\") pod \"de7c4c4d-2e02-4c99-b9e0-8f15391d7871\" (UID: \"de7c4c4d-2e02-4c99-b9e0-8f15391d7871\") "
Nov 03 20:34:43.633886 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.633857    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-plugins-config-volume" pod "de7c4c4d-2e02-4c99-b9e0-8f15391d7871" (UID: "de7c4c4d-2e02-4c99-b9e0-8f15391d7871"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:43.636123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.636095    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-kube-api-access-bb86n" pod "de7c4c4d-2e02-4c99-b9e0-8f15391d7871" (UID: "de7c4c4d-2e02-4c99-b9e0-8f15391d7871"). InnerVolumeSpecName "kube-api-access-bb86n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:43.736197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.736157    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb86n\" (UniqueName: \"kubernetes.io/projected/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-kube-api-access-bb86n\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:43.736197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.736195    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de7c4c4d-2e02-4c99-b9e0-8f15391d7871-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:43.807426 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.807384    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:43.933645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.933535    5659 generic.go:356] "Generic (PLEG): container finished" podID="de7c4c4d-2e02-4c99-b9e0-8f15391d7871" containerID="e371e3e73b5059592ffdc3fbc899049cb319e3e96b864bba71471a04e5931da7" exitCode=0
Nov 03 20:34:43.933645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.933595    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"
Nov 03 20:34:43.933645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.933625    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7" event={"ID":"de7c4c4d-2e02-4c99-b9e0-8f15391d7871","Type":"ContainerDied","Data":"e371e3e73b5059592ffdc3fbc899049cb319e3e96b864bba71471a04e5931da7"}
Nov 03 20:34:43.933939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.933682    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7" event={"ID":"de7c4c4d-2e02-4c99-b9e0-8f15391d7871","Type":"ContainerDied","Data":"8dd7618243d63736eb5e1799ac284bc444d5fb9a5f7c9f0f45bc40f3f8e6026b"}
Nov 03 20:34:43.933939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.933713    5659 scope.go:122] "RemoveContainer" containerID="e371e3e73b5059592ffdc3fbc899049cb319e3e96b864bba71471a04e5931da7"
Nov 03 20:34:43.947790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.947766    5659 scope.go:122] "RemoveContainer" containerID="e371e3e73b5059592ffdc3fbc899049cb319e3e96b864bba71471a04e5931da7"
Nov 03 20:34:43.948102 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:34:43.948081    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e371e3e73b5059592ffdc3fbc899049cb319e3e96b864bba71471a04e5931da7\": container with ID starting with e371e3e73b5059592ffdc3fbc899049cb319e3e96b864bba71471a04e5931da7 not found: ID does not exist" containerID="e371e3e73b5059592ffdc3fbc899049cb319e3e96b864bba71471a04e5931da7"
Nov 03 20:34:43.948166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.948114    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e371e3e73b5059592ffdc3fbc899049cb319e3e96b864bba71471a04e5931da7"} err="failed to get container status \"e371e3e73b5059592ffdc3fbc899049cb319e3e96b864bba71471a04e5931da7\": rpc error: code = NotFound desc = could not find container \"e371e3e73b5059592ffdc3fbc899049cb319e3e96b864bba71471a04e5931da7\": container with ID starting with e371e3e73b5059592ffdc3fbc899049cb319e3e96b864bba71471a04e5931da7 not found: ID does not exist"
Nov 03 20:34:43.960402 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.960372    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"]
Nov 03 20:34:43.977700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:43.977660    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-98529d-84f599d4f5-dxgp7"]
Nov 03 20:34:44.635879 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:44.635840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:44.925703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:44.925608    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:45.051008 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:45.050973    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:45.222528 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:34:45.222475    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:34:45.335336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:45.335300    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:45.975425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:45.975388    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7c4c4d-2e02-4c99-b9e0-8f15391d7871" path="/var/lib/kubelet/pods/de7c4c4d-2e02-4c99-b9e0-8f15391d7871/volumes"
Nov 03 20:34:46.124503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:46.123511    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:46.370562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:46.370425    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:46.764958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:46.764925    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:47.437791 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:47.437754    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:48.122315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:48.122266    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:48.849153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:48.849115    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:49.538876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:49.538838    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:50.393766 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:50.392983    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:51.235268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:51.235235    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:51.763384 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:51.763342    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"]
Nov 03 20:34:51.767937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:51.767899    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5" podUID="80e7029d-ce9c-452a-8d22-2b7b12212c7c" containerName="maas-api" containerID="cri-o://31d4610d3e4a3d53c5fd2dbecec267841d9fa3e75680920c95140138a79e68de" gracePeriod=30
Nov 03 20:34:51.829949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:51.829912    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:52.172465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.172431    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:52.191400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.191123    5659 generic.go:356] "Generic (PLEG): container finished" podID="80e7029d-ce9c-452a-8d22-2b7b12212c7c" containerID="31d4610d3e4a3d53c5fd2dbecec267841d9fa3e75680920c95140138a79e68de" exitCode=0
Nov 03 20:34:52.191400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.191322    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5" event={"ID":"80e7029d-ce9c-452a-8d22-2b7b12212c7c","Type":"ContainerDied","Data":"31d4610d3e4a3d53c5fd2dbecec267841d9fa3e75680920c95140138a79e68de"}
Nov 03 20:34:52.720163 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.720129    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:34:52.749450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.749409    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-metrics-certs\") pod \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") "
Nov 03 20:34:52.751924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.749558    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e7029d-ce9c-452a-8d22-2b7b12212c7c-kube-api-access-5vvgq\" (UniqueName: \"kubernetes.io/projected/80e7029d-ce9c-452a-8d22-2b7b12212c7c-kube-api-access-5vvgq\") pod \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") "
Nov 03 20:34:52.751924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.749779    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-maas-api-tls\") pod \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") "
Nov 03 20:34:52.751924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.749854    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e7029d-ce9c-452a-8d22-2b7b12212c7c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e7029d-ce9c-452a-8d22-2b7b12212c7c-openshift-service-ca\") pod \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\" (UID: \"80e7029d-ce9c-452a-8d22-2b7b12212c7c\") "
Nov 03 20:34:52.751924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.750351    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e7029d-ce9c-452a-8d22-2b7b12212c7c-openshift-service-ca" pod "80e7029d-ce9c-452a-8d22-2b7b12212c7c" (UID: "80e7029d-ce9c-452a-8d22-2b7b12212c7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:52.751924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.750636    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e7029d-ce9c-452a-8d22-2b7b12212c7c-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:52.752736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.752708    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-maas-api-tls" pod "80e7029d-ce9c-452a-8d22-2b7b12212c7c" (UID: "80e7029d-ce9c-452a-8d22-2b7b12212c7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:34:52.753135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.753115    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-metrics-certs" pod "80e7029d-ce9c-452a-8d22-2b7b12212c7c" (UID: "80e7029d-ce9c-452a-8d22-2b7b12212c7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:34:52.754007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.753946    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e7029d-ce9c-452a-8d22-2b7b12212c7c-kube-api-access-5vvgq" pod "80e7029d-ce9c-452a-8d22-2b7b12212c7c" (UID: "80e7029d-ce9c-452a-8d22-2b7b12212c7c"). InnerVolumeSpecName "kube-api-access-5vvgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:52.774585 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.774464    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:52.851952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.851870    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:52.851952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.851901    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vvgq\" (UniqueName: \"kubernetes.io/projected/80e7029d-ce9c-452a-8d22-2b7b12212c7c-kube-api-access-5vvgq\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:52.851952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:52.851912    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e7029d-ce9c-452a-8d22-2b7b12212c7c-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:53.201407 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:53.201322    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"
Nov 03 20:34:53.201407 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:53.201360    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5" event={"ID":"80e7029d-ce9c-452a-8d22-2b7b12212c7c","Type":"ContainerDied","Data":"1bc6b310615d06852c30fcd528b518aa36428b9b7b198ac619d766fedbedd9bc"}
Nov 03 20:34:53.201699 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:53.201422    5659 scope.go:122] "RemoveContainer" containerID="31d4610d3e4a3d53c5fd2dbecec267841d9fa3e75680920c95140138a79e68de"
Nov 03 20:34:53.228185 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:53.228158    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"]
Nov 03 20:34:53.235454 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:53.235406    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-98529d-85bb678d6-jqrs5"]
Nov 03 20:34:53.492503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:53.490067    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:53.753624 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:53.753497    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:53.975306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:53.975257    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e7029d-ce9c-452a-8d22-2b7b12212c7c" path="/var/lib/kubelet/pods/80e7029d-ce9c-452a-8d22-2b7b12212c7c/volumes"
Nov 03 20:34:54.089366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:54.089257    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:54.351627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:54.351526    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:54.671396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:54.671272    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:54.851582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:54.851523    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:55.812605 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:55.804455    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:56.422874 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:56.422840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:57.033333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:57.033300    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:57.230081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:57.230050    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:57.868388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:57.868345    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:58.177558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:58.177142    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:58.398505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:58.396100    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:58.621010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:58.620818    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:59.337090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:59.337006    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:59.553505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:34:59.553455    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:00.248993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:00.248942    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:01.062887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:01.062755    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:01.733046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:01.733009    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:01.876419 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:01.876381    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:02.079559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:02.079390    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:02.926979 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:02.926948    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:03.491940 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:03.491906    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:04.046096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:04.046059    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:04.410145 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:04.410046    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"]
Nov 03 20:35:04.510516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:04.510459    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:04.649945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:04.649906    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:05.053423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:05.053389    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:05.071950 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:05.071912    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"]
Nov 03 20:35:05.310628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:05.305903    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:05.672888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:05.672780    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:06.202538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:06.202446    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:06.336727 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:06.333326    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:06.862659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:06.862619    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:06.997027 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:06.996989    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:07.364938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:07.364904    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:07.808263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:07.808215    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:08.053550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:08.024174    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:08.819348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:08.819316    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:09.413313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:09.413268    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst" podUID="a351e8e7-b09d-44fd-ba3e-8c811db79f8a" containerName="payload-processing" containerID="cri-o://5622f72b2bcef470edb4281fefb6316e09ca2c2931c2390b5632b5e452628d32" gracePeriod=25
Nov 03 20:35:09.634588 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:09.634078    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:09.859107 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:09.859063    5659 generic.go:356] "Generic (PLEG): container finished" podID="a351e8e7-b09d-44fd-ba3e-8c811db79f8a" containerID="5622f72b2bcef470edb4281fefb6316e09ca2c2931c2390b5632b5e452628d32" exitCode=0
Nov 03 20:35:09.859401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:09.859264    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst" event={"ID":"a351e8e7-b09d-44fd-ba3e-8c811db79f8a","Type":"ContainerDied","Data":"5622f72b2bcef470edb4281fefb6316e09ca2c2931c2390b5632b5e452628d32"}
Nov 03 20:35:09.859401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:09.859361    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst" event={"ID":"a351e8e7-b09d-44fd-ba3e-8c811db79f8a","Type":"ContainerDied","Data":"7c5361a6e26962ec8e0a4804e8c569f5cf32e0661016cf4a9a2740fc52fe7d46"}
Nov 03 20:35:09.859401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:09.859400    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c5361a6e26962ec8e0a4804e8c569f5cf32e0661016cf4a9a2740fc52fe7d46"
Nov 03 20:35:09.867208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:09.867183    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"
Nov 03 20:35:10.008674 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.008633    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-plugins-config-volume\") pod \"a351e8e7-b09d-44fd-ba3e-8c811db79f8a\" (UID: \"a351e8e7-b09d-44fd-ba3e-8c811db79f8a\") "
Nov 03 20:35:10.008991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.008728    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-kube-api-access-lm9df\" (UniqueName: \"kubernetes.io/projected/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-kube-api-access-lm9df\") pod \"a351e8e7-b09d-44fd-ba3e-8c811db79f8a\" (UID: \"a351e8e7-b09d-44fd-ba3e-8c811db79f8a\") "
Nov 03 20:35:10.009066 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.009002    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-plugins-config-volume" pod "a351e8e7-b09d-44fd-ba3e-8c811db79f8a" (UID: "a351e8e7-b09d-44fd-ba3e-8c811db79f8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:10.009438 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.009417    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:10.011059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.011032    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-kube-api-access-lm9df" pod "a351e8e7-b09d-44fd-ba3e-8c811db79f8a" (UID: "a351e8e7-b09d-44fd-ba3e-8c811db79f8a"). InnerVolumeSpecName "kube-api-access-lm9df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:10.073667 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.073564    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl" podUID="4d5dac14-76e3-44c6-9926-44b0156716c7" containerName="payload-pre-processing" containerID="cri-o://5b8fd1903abbe54fde7503fb95a9b5d3030ce6ae3a762b3e65857918473c43c4" gracePeriod=25
Nov 03 20:35:10.111790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.111762    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm9df\" (UniqueName: \"kubernetes.io/projected/a351e8e7-b09d-44fd-ba3e-8c811db79f8a-kube-api-access-lm9df\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:10.308745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.308696    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:10.519190 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.519164    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"
Nov 03 20:35:10.627754 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.627722    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d5dac14-76e3-44c6-9926-44b0156716c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5dac14-76e3-44c6-9926-44b0156716c7-plugins-config-volume\") pod \"4d5dac14-76e3-44c6-9926-44b0156716c7\" (UID: \"4d5dac14-76e3-44c6-9926-44b0156716c7\") "
Nov 03 20:35:10.627955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.627878    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5dac14-76e3-44c6-9926-44b0156716c7-kube-api-access-bdr6t\" (UniqueName: \"kubernetes.io/projected/4d5dac14-76e3-44c6-9926-44b0156716c7-kube-api-access-bdr6t\") pod \"4d5dac14-76e3-44c6-9926-44b0156716c7\" (UID: \"4d5dac14-76e3-44c6-9926-44b0156716c7\") "
Nov 03 20:35:10.628189 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.628164    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5dac14-76e3-44c6-9926-44b0156716c7-plugins-config-volume" pod "4d5dac14-76e3-44c6-9926-44b0156716c7" (UID: "4d5dac14-76e3-44c6-9926-44b0156716c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:10.629098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.629070    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5dac14-76e3-44c6-9926-44b0156716c7-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:10.630626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.630599    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5dac14-76e3-44c6-9926-44b0156716c7-kube-api-access-bdr6t" pod "4d5dac14-76e3-44c6-9926-44b0156716c7" (UID: "4d5dac14-76e3-44c6-9926-44b0156716c7"). InnerVolumeSpecName "kube-api-access-bdr6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:10.731194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.731114    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdr6t\" (UniqueName: \"kubernetes.io/projected/4d5dac14-76e3-44c6-9926-44b0156716c7-kube-api-access-bdr6t\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:10.870594 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.870556    5659 generic.go:356] "Generic (PLEG): container finished" podID="4d5dac14-76e3-44c6-9926-44b0156716c7" containerID="5b8fd1903abbe54fde7503fb95a9b5d3030ce6ae3a762b3e65857918473c43c4" exitCode=0
Nov 03 20:35:10.870787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.870602    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"
Nov 03 20:35:10.870787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.870642    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl" event={"ID":"4d5dac14-76e3-44c6-9926-44b0156716c7","Type":"ContainerDied","Data":"5b8fd1903abbe54fde7503fb95a9b5d3030ce6ae3a762b3e65857918473c43c4"}
Nov 03 20:35:10.870787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.870701    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl" event={"ID":"4d5dac14-76e3-44c6-9926-44b0156716c7","Type":"ContainerDied","Data":"b700f3a4e41357f42eb26139ec34a191dc0f3347b11ffa9f36570759fda7e028"}
Nov 03 20:35:10.870787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.870729    5659 scope.go:122] "RemoveContainer" containerID="5b8fd1903abbe54fde7503fb95a9b5d3030ce6ae3a762b3e65857918473c43c4"
Nov 03 20:35:10.871060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.871043    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"
Nov 03 20:35:10.884855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.884832    5659 scope.go:122] "RemoveContainer" containerID="5b8fd1903abbe54fde7503fb95a9b5d3030ce6ae3a762b3e65857918473c43c4"
Nov 03 20:35:10.885288 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:35:10.885135    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8fd1903abbe54fde7503fb95a9b5d3030ce6ae3a762b3e65857918473c43c4\": container with ID starting with 5b8fd1903abbe54fde7503fb95a9b5d3030ce6ae3a762b3e65857918473c43c4 not found: ID does not exist" containerID="5b8fd1903abbe54fde7503fb95a9b5d3030ce6ae3a762b3e65857918473c43c4"
Nov 03 20:35:10.885288 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.885172    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8fd1903abbe54fde7503fb95a9b5d3030ce6ae3a762b3e65857918473c43c4"} err="failed to get container status \"5b8fd1903abbe54fde7503fb95a9b5d3030ce6ae3a762b3e65857918473c43c4\": rpc error: code = NotFound desc = could not find container \"5b8fd1903abbe54fde7503fb95a9b5d3030ce6ae3a762b3e65857918473c43c4\": container with ID starting with 5b8fd1903abbe54fde7503fb95a9b5d3030ce6ae3a762b3e65857918473c43c4 not found: ID does not exist"
Nov 03 20:35:10.904990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.904955    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"]
Nov 03 20:35:10.910284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.910257    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-74dfc6-7cc66559cf-cgkbl"]
Nov 03 20:35:10.920088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.920057    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"]
Nov 03 20:35:10.924119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:10.924093    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b503a8-7bf9b44576-cxrst"]
Nov 03 20:35:11.139567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:11.120916    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:11.897549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:11.897515    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:11.975204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:11.975168    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5dac14-76e3-44c6-9926-44b0156716c7" path="/var/lib/kubelet/pods/4d5dac14-76e3-44c6-9926-44b0156716c7/volumes"
Nov 03 20:35:11.975741 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:11.975723    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a351e8e7-b09d-44fd-ba3e-8c811db79f8a" path="/var/lib/kubelet/pods/a351e8e7-b09d-44fd-ba3e-8c811db79f8a/volumes"
Nov 03 20:35:12.270682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:12.270647    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:12.531764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:12.531665    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:13.136410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:13.136354    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:13.365505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:13.356249    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:13.504509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:13.504143    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:13.890650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:13.890562    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:14.204817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:14.204783    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"]
Nov 03 20:35:14.205238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:14.205048    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy" containerID="cri-o://5bfd63224718fa4e97f9fd1819aaa34ddfb5a5342005c3c5e695e94a05ace7bc" gracePeriod=30
Nov 03 20:35:14.241118 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:14.241083    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:14.604582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:14.600021    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:15.127632 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:15.127592    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:15.218927 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:35:15.218888    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:35:15.331125 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:15.331084    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:16.146194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:16.146156    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:16.925055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:16.925000    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:17.620331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:17.620296    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:18.284734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:18.284202    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:18.901430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:18.901394    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:19.041104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.041067    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:19.190338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.190251    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:19.355095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.355061    5659 patch_prober.go:29] interesting pod/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:35:19.355522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.355130    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 03 20:35:19.491202 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.491159    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:19.631264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.631227    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:35:19.688702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.688669    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-credential-socket\") pod \"fdffad9b-17e0-408b-b096-73ca58b439b3\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") "
Nov 03 20:35:19.688922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.688868    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-kube-api-access-bslwm\" (UniqueName: \"kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-kube-api-access-bslwm\") pod \"fdffad9b-17e0-408b-b096-73ca58b439b3\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") "
Nov 03 20:35:19.688998 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.688924    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-podinfo\") pod \"fdffad9b-17e0-408b-b096-73ca58b439b3\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") "
Nov 03 20:35:19.689052 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.689008    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-envoy\") pod \"fdffad9b-17e0-408b-b096-73ca58b439b3\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") "
Nov 03 20:35:19.689628 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.689590    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-credential-socket" pod "fdffad9b-17e0-408b-b096-73ca58b439b3" (UID: "fdffad9b-17e0-408b-b096-73ca58b439b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:19.690082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.690059    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-socket" pod "fdffad9b-17e0-408b-b096-73ca58b439b3" (UID: "fdffad9b-17e0-408b-b096-73ca58b439b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:19.690241 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.689074    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-socket\") pod \"fdffad9b-17e0-408b-b096-73ca58b439b3\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") "
Nov 03 20:35:19.690340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.690304    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-data\") pod \"fdffad9b-17e0-408b-b096-73ca58b439b3\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") "
Nov 03 20:35:19.690399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.690354    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-certs\") pod \"fdffad9b-17e0-408b-b096-73ca58b439b3\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") "
Nov 03 20:35:19.690459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.690404    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdffad9b-17e0-408b-b096-73ca58b439b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdffad9b-17e0-408b-b096-73ca58b439b3-istiod-ca-cert\") pod \"fdffad9b-17e0-408b-b096-73ca58b439b3\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") "
Nov 03 20:35:19.690459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.690447    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-token\" (UniqueName: \"kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-token\") pod \"fdffad9b-17e0-408b-b096-73ca58b439b3\" (UID: \"fdffad9b-17e0-408b-b096-73ca58b439b3\") "
Nov 03 20:35:19.691412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.690799    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-data" pod "fdffad9b-17e0-408b-b096-73ca58b439b3" (UID: "fdffad9b-17e0-408b-b096-73ca58b439b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:19.691412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.691352    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:19.691412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.691377    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:19.691412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.691374    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdffad9b-17e0-408b-b096-73ca58b439b3-istiod-ca-cert" pod "fdffad9b-17e0-408b-b096-73ca58b439b3" (UID: "fdffad9b-17e0-408b-b096-73ca58b439b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:19.691412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.691391    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:19.694836 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.694112    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-token" pod "fdffad9b-17e0-408b-b096-73ca58b439b3" (UID: "fdffad9b-17e0-408b-b096-73ca58b439b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:19.695205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.695185    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-certs" pod "fdffad9b-17e0-408b-b096-73ca58b439b3" (UID: "fdffad9b-17e0-408b-b096-73ca58b439b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:19.696389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.696361    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-kube-api-access-bslwm" pod "fdffad9b-17e0-408b-b096-73ca58b439b3" (UID: "fdffad9b-17e0-408b-b096-73ca58b439b3"). InnerVolumeSpecName "kube-api-access-bslwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:19.700559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.698458    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-podinfo" pod "fdffad9b-17e0-408b-b096-73ca58b439b3" (UID: "fdffad9b-17e0-408b-b096-73ca58b439b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:35:19.700790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.700760    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-envoy" pod "fdffad9b-17e0-408b-b096-73ca58b439b3" (UID: "fdffad9b-17e0-408b-b096-73ca58b439b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:19.794045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.793945    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bslwm\" (UniqueName: \"kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-kube-api-access-bslwm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:19.794045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.793986    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:19.794045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.794003    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:19.794045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.794017    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdffad9b-17e0-408b-b096-73ca58b439b3-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:19.794045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.794031    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdffad9b-17e0-408b-b096-73ca58b439b3-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:19.794045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.794044    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdffad9b-17e0-408b-b096-73ca58b439b3-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:19.795189 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.795159    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"]
Nov 03 20:35:19.795412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.795388    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt" podUID="1181f6f2-8a83-4b5f-bf64-77a556de321d" containerName="istio-proxy" containerID="cri-o://d96f5ec7128d6df2a716e894e3a4facc7b2e0200e28ef9e52099f97d237a6bdc" gracePeriod=30
Nov 03 20:35:19.845527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:19.845467    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:20.142098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.141993    5659 generic.go:356] "Generic (PLEG): container finished" podID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerID="5bfd63224718fa4e97f9fd1819aaa34ddfb5a5342005c3c5e695e94a05ace7bc" exitCode=0
Nov 03 20:35:20.143077 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.143055    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"
Nov 03 20:35:20.159506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.147919    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" event={"ID":"fdffad9b-17e0-408b-b096-73ca58b439b3","Type":"ContainerDied","Data":"5bfd63224718fa4e97f9fd1819aaa34ddfb5a5342005c3c5e695e94a05ace7bc"}
Nov 03 20:35:20.160994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.160967    5659 scope.go:122] "RemoveContainer" containerID="5bfd63224718fa4e97f9fd1819aaa34ddfb5a5342005c3c5e695e94a05ace7bc"
Nov 03 20:35:20.161852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.161824    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx" event={"ID":"fdffad9b-17e0-408b-b096-73ca58b439b3","Type":"ContainerDied","Data":"0fd896136965acafe1ae1f76000a5636a3799b1d0dbe3f1b0f423f429f53b3c5"}
Nov 03 20:35:20.233540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.233503    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"]
Nov 03 20:35:20.240119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.240084    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:20.245973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.245929    5659 scope.go:122] "RemoveContainer" containerID="30e83865b7ebf30fc3a0e9227f984d63c3fb78162ab1d055f998b6b84f917ead"
Nov 03 20:35:20.249092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.249067    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-74dfc6-openshift-default-bc45d6cb9-fxvzx"]
Nov 03 20:35:20.344337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.342809    5659 scope.go:122] "RemoveContainer" containerID="5bfd63224718fa4e97f9fd1819aaa34ddfb5a5342005c3c5e695e94a05ace7bc"
Nov 03 20:35:20.344337 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:35:20.343576    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfd63224718fa4e97f9fd1819aaa34ddfb5a5342005c3c5e695e94a05ace7bc\": container with ID starting with 5bfd63224718fa4e97f9fd1819aaa34ddfb5a5342005c3c5e695e94a05ace7bc not found: ID does not exist" containerID="5bfd63224718fa4e97f9fd1819aaa34ddfb5a5342005c3c5e695e94a05ace7bc"
Nov 03 20:35:20.344337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.343631    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfd63224718fa4e97f9fd1819aaa34ddfb5a5342005c3c5e695e94a05ace7bc"} err="failed to get container status \"5bfd63224718fa4e97f9fd1819aaa34ddfb5a5342005c3c5e695e94a05ace7bc\": rpc error: code = NotFound desc = could not find container \"5bfd63224718fa4e97f9fd1819aaa34ddfb5a5342005c3c5e695e94a05ace7bc\": container with ID starting with 5bfd63224718fa4e97f9fd1819aaa34ddfb5a5342005c3c5e695e94a05ace7bc not found: ID does not exist"
Nov 03 20:35:20.344337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.343660    5659 scope.go:122] "RemoveContainer" containerID="30e83865b7ebf30fc3a0e9227f984d63c3fb78162ab1d055f998b6b84f917ead"
Nov 03 20:35:20.356528 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:35:20.353707    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e83865b7ebf30fc3a0e9227f984d63c3fb78162ab1d055f998b6b84f917ead\": container with ID starting with 30e83865b7ebf30fc3a0e9227f984d63c3fb78162ab1d055f998b6b84f917ead not found: ID does not exist" containerID="30e83865b7ebf30fc3a0e9227f984d63c3fb78162ab1d055f998b6b84f917ead"
Nov 03 20:35:20.356528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.353757    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e83865b7ebf30fc3a0e9227f984d63c3fb78162ab1d055f998b6b84f917ead"} err="failed to get container status \"30e83865b7ebf30fc3a0e9227f984d63c3fb78162ab1d055f998b6b84f917ead\": rpc error: code = NotFound desc = could not find container \"30e83865b7ebf30fc3a0e9227f984d63c3fb78162ab1d055f998b6b84f917ead\": container with ID starting with 30e83865b7ebf30fc3a0e9227f984d63c3fb78162ab1d055f998b6b84f917ead not found: ID does not exist"
Nov 03 20:35:20.652598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:20.652561    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:21.028328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:21.028294    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:21.423996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:21.423906    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:21.975198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:21.975156    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" path="/var/lib/kubelet/pods/fdffad9b-17e0-408b-b096-73ca58b439b3/volumes"
Nov 03 20:35:22.187507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:22.187017    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:22.341546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:22.341188    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:22.468977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:22.468944    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:22.793546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:22.785328    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:23.145463 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:23.145377    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:23.821391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:23.821351    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:24.893318 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:24.893255    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:25.192821 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.192793    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:35:25.257477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.257437    5659 generic.go:356] "Generic (PLEG): container finished" podID="1181f6f2-8a83-4b5f-bf64-77a556de321d" containerID="d96f5ec7128d6df2a716e894e3a4facc7b2e0200e28ef9e52099f97d237a6bdc" exitCode=0
Nov 03 20:35:25.258507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.257752    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"
Nov 03 20:35:25.258507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.257772    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt" event={"ID":"1181f6f2-8a83-4b5f-bf64-77a556de321d","Type":"ContainerDied","Data":"d96f5ec7128d6df2a716e894e3a4facc7b2e0200e28ef9e52099f97d237a6bdc"}
Nov 03 20:35:25.258507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.258346    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt" event={"ID":"1181f6f2-8a83-4b5f-bf64-77a556de321d","Type":"ContainerDied","Data":"95144b6cbb2ab16e63d8ea80d702e5a1ee260d48629451963bb392c204acddc2"}
Nov 03 20:35:25.258507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.258382    5659 scope.go:122] "RemoveContainer" containerID="d96f5ec7128d6df2a716e894e3a4facc7b2e0200e28ef9e52099f97d237a6bdc"
Nov 03 20:35:25.272922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.272898    5659 scope.go:122] "RemoveContainer" containerID="d96f5ec7128d6df2a716e894e3a4facc7b2e0200e28ef9e52099f97d237a6bdc"
Nov 03 20:35:25.273253 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:35:25.273226    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d96f5ec7128d6df2a716e894e3a4facc7b2e0200e28ef9e52099f97d237a6bdc\": container with ID starting with d96f5ec7128d6df2a716e894e3a4facc7b2e0200e28ef9e52099f97d237a6bdc not found: ID does not exist" containerID="d96f5ec7128d6df2a716e894e3a4facc7b2e0200e28ef9e52099f97d237a6bdc"
Nov 03 20:35:25.273306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.273268    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96f5ec7128d6df2a716e894e3a4facc7b2e0200e28ef9e52099f97d237a6bdc"} err="failed to get container status \"d96f5ec7128d6df2a716e894e3a4facc7b2e0200e28ef9e52099f97d237a6bdc\": rpc error: code = NotFound desc = could not find container \"d96f5ec7128d6df2a716e894e3a4facc7b2e0200e28ef9e52099f97d237a6bdc\": container with ID starting with d96f5ec7128d6df2a716e894e3a4facc7b2e0200e28ef9e52099f97d237a6bdc not found: ID does not exist"
Nov 03 20:35:25.295835 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.295796    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1181f6f2-8a83-4b5f-bf64-77a556de321d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1181f6f2-8a83-4b5f-bf64-77a556de321d-istiod-ca-cert\") pod \"1181f6f2-8a83-4b5f-bf64-77a556de321d\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") "
Nov 03 20:35:25.296010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.295876    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-podinfo\") pod \"1181f6f2-8a83-4b5f-bf64-77a556de321d\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") "
Nov 03 20:35:25.296096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.296020    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-kube-api-access-nxf62\" (UniqueName: \"kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-kube-api-access-nxf62\") pod \"1181f6f2-8a83-4b5f-bf64-77a556de321d\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") "
Nov 03 20:35:25.296162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.296096    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-credential-socket\") pod \"1181f6f2-8a83-4b5f-bf64-77a556de321d\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") "
Nov 03 20:35:25.296228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.296171    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-envoy\") pod \"1181f6f2-8a83-4b5f-bf64-77a556de321d\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") "
Nov 03 20:35:25.296291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.296258    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1181f6f2-8a83-4b5f-bf64-77a556de321d-istiod-ca-cert" pod "1181f6f2-8a83-4b5f-bf64-77a556de321d" (UID: "1181f6f2-8a83-4b5f-bf64-77a556de321d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:25.296353 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.296280    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-data\") pod \"1181f6f2-8a83-4b5f-bf64-77a556de321d\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") "
Nov 03 20:35:25.296470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.296450    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-token\" (UniqueName: \"kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-token\") pod \"1181f6f2-8a83-4b5f-bf64-77a556de321d\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") "
Nov 03 20:35:25.297418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.296625    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-data" pod "1181f6f2-8a83-4b5f-bf64-77a556de321d" (UID: "1181f6f2-8a83-4b5f-bf64-77a556de321d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:25.297418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.296530    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-certs\") pod \"1181f6f2-8a83-4b5f-bf64-77a556de321d\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") "
Nov 03 20:35:25.297418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.296814    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-socket\") pod \"1181f6f2-8a83-4b5f-bf64-77a556de321d\" (UID: \"1181f6f2-8a83-4b5f-bf64-77a556de321d\") "
Nov 03 20:35:25.297418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.297023    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-certs" pod "1181f6f2-8a83-4b5f-bf64-77a556de321d" (UID: "1181f6f2-8a83-4b5f-bf64-77a556de321d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:25.297418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.297030    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-credential-socket" pod "1181f6f2-8a83-4b5f-bf64-77a556de321d" (UID: "1181f6f2-8a83-4b5f-bf64-77a556de321d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:25.297418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.297314    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-socket" pod "1181f6f2-8a83-4b5f-bf64-77a556de321d" (UID: "1181f6f2-8a83-4b5f-bf64-77a556de321d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:25.297904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.297787    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:25.297904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.297811    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:25.297904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.297827    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:25.297904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.297841    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:25.297904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.297855    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1181f6f2-8a83-4b5f-bf64-77a556de321d-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:25.300251 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.300182    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-podinfo" pod "1181f6f2-8a83-4b5f-bf64-77a556de321d" (UID: "1181f6f2-8a83-4b5f-bf64-77a556de321d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:35:25.300615 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.300590    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-token" pod "1181f6f2-8a83-4b5f-bf64-77a556de321d" (UID: "1181f6f2-8a83-4b5f-bf64-77a556de321d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:25.301668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.301644    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-kube-api-access-nxf62" pod "1181f6f2-8a83-4b5f-bf64-77a556de321d" (UID: "1181f6f2-8a83-4b5f-bf64-77a556de321d"). InnerVolumeSpecName "kube-api-access-nxf62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:25.302103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.302081    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-envoy" pod "1181f6f2-8a83-4b5f-bf64-77a556de321d" (UID: "1181f6f2-8a83-4b5f-bf64-77a556de321d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:25.400530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.400437    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:25.400530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.400495    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:25.400530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.400514    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxf62\" (UniqueName: \"kubernetes.io/projected/1181f6f2-8a83-4b5f-bf64-77a556de321d-kube-api-access-nxf62\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:25.400530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.400530    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1181f6f2-8a83-4b5f-bf64-77a556de321d-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:25.545954 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.545920    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:25.589967 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.589926    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"]
Nov 03 20:35:25.592187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.592159    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b503a8-openshift-default-69548545d9-rb5vt"]
Nov 03 20:35:25.974987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:25.974948    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1181f6f2-8a83-4b5f-bf64-77a556de321d" path="/var/lib/kubelet/pods/1181f6f2-8a83-4b5f-bf64-77a556de321d/volumes"
Nov 03 20:35:26.321027 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:26.320939    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:26.465507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:26.465435    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:27.263944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:27.263905    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:27.853059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:27.853023    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:28.137327 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:28.137198    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:28.802508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:28.798347    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:29.036103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:29.036067    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:29.658744 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:29.658710    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:30.375902 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:30.375868    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:30.512584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:30.512541    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:31.065137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:31.065097    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:31.855467 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:31.855430    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:32.322388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:32.322349    5659 scope.go:122] "RemoveContainer" containerID="5622f72b2bcef470edb4281fefb6316e09ca2c2931c2390b5632b5e452628d32"
Nov 03 20:35:32.614456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:32.614367    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:33.298243 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:33.298206    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:33.475516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:33.475091    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:34.333599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:34.333563    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:34.484383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:34.484343    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:35.164425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:35.164323    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:35.821047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:35.821016    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:36.525864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:36.525830    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:36.996945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:36.996909    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"]
Nov 03 20:35:36.997440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:36.997165    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f" podUID="8aca7ab0-c961-4029-9d38-5e5697f39f76" containerName="maas-api" containerID="cri-o://bfa4b3cdfc4796a833d51fb56de4a50c977da89c403d2ccf8e27d77f89597ebb" gracePeriod=30
Nov 03 20:35:37.073032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.073005    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:37.408845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.408751    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:37.675565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.675400    5659 generic.go:356] "Generic (PLEG): container finished" podID="8aca7ab0-c961-4029-9d38-5e5697f39f76" containerID="bfa4b3cdfc4796a833d51fb56de4a50c977da89c403d2ccf8e27d77f89597ebb" exitCode=0
Nov 03 20:35:37.675798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.675777    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f" event={"ID":"8aca7ab0-c961-4029-9d38-5e5697f39f76","Type":"ContainerDied","Data":"bfa4b3cdfc4796a833d51fb56de4a50c977da89c403d2ccf8e27d77f89597ebb"}
Nov 03 20:35:37.737158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.737118    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:35:37.759979 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.759947    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aca7ab0-c961-4029-9d38-5e5697f39f76-kube-api-access-qb2qg\" (UniqueName: \"kubernetes.io/projected/8aca7ab0-c961-4029-9d38-5e5697f39f76-kube-api-access-qb2qg\") pod \"8aca7ab0-c961-4029-9d38-5e5697f39f76\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") "
Nov 03 20:35:37.760255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.760231    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-metrics-certs\") pod \"8aca7ab0-c961-4029-9d38-5e5697f39f76\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") "
Nov 03 20:35:37.760589 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.760568    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aca7ab0-c961-4029-9d38-5e5697f39f76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aca7ab0-c961-4029-9d38-5e5697f39f76-openshift-service-ca\") pod \"8aca7ab0-c961-4029-9d38-5e5697f39f76\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") "
Nov 03 20:35:37.761153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.761032    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aca7ab0-c961-4029-9d38-5e5697f39f76-openshift-service-ca" pod "8aca7ab0-c961-4029-9d38-5e5697f39f76" (UID: "8aca7ab0-c961-4029-9d38-5e5697f39f76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:37.761631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.761611    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-maas-api-tls\") pod \"8aca7ab0-c961-4029-9d38-5e5697f39f76\" (UID: \"8aca7ab0-c961-4029-9d38-5e5697f39f76\") "
Nov 03 20:35:37.764029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.764007    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aca7ab0-c961-4029-9d38-5e5697f39f76-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:37.766060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.764322    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aca7ab0-c961-4029-9d38-5e5697f39f76-kube-api-access-qb2qg" pod "8aca7ab0-c961-4029-9d38-5e5697f39f76" (UID: "8aca7ab0-c961-4029-9d38-5e5697f39f76"). InnerVolumeSpecName "kube-api-access-qb2qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:37.766791 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.766767    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-metrics-certs" pod "8aca7ab0-c961-4029-9d38-5e5697f39f76" (UID: "8aca7ab0-c961-4029-9d38-5e5697f39f76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:35:37.768570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.768207    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-maas-api-tls" pod "8aca7ab0-c961-4029-9d38-5e5697f39f76" (UID: "8aca7ab0-c961-4029-9d38-5e5697f39f76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:35:37.884343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.884300    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:37.884343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.884340    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb2qg\" (UniqueName: \"kubernetes.io/projected/8aca7ab0-c961-4029-9d38-5e5697f39f76-kube-api-access-qb2qg\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:37.884695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:37.884358    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aca7ab0-c961-4029-9d38-5e5697f39f76-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:38.148723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:38.148676    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:38.685821 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:38.685786    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f" event={"ID":"8aca7ab0-c961-4029-9d38-5e5697f39f76","Type":"ContainerDied","Data":"c73daa4a6ac38ab30cda44869a103e38c61df0c6004d68830a1a7939dc111fa0"}
Nov 03 20:35:38.686040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:38.685848    5659 scope.go:122] "RemoveContainer" containerID="bfa4b3cdfc4796a833d51fb56de4a50c977da89c403d2ccf8e27d77f89597ebb"
Nov 03 20:35:38.686040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:38.685802    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"
Nov 03 20:35:38.707029 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:38.707003    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"]
Nov 03 20:35:38.711354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:38.711328    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a0d4ec-6dcdd75476-ztm6f"]
Nov 03 20:35:38.850004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:38.849962    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:39.527606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:39.527573    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:39.972415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:39.972380    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aca7ab0-c961-4029-9d38-5e5697f39f76" path="/var/lib/kubelet/pods/8aca7ab0-c961-4029-9d38-5e5697f39f76/volumes"
Nov 03 20:35:40.228960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:40.228868    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:40.899857 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:40.899827    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:41.025252 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:41.025189    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:41.438142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:41.438111    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:41.793644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:41.793609    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:42.075880 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:42.075775    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:42.575689 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:42.575654    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:42.758629 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:42.758595    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:43.496510 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:43.492214    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:44.333478 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:44.333436    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:45.028188 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:45.028155    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:45.220014 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:35:45.219975    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:35:45.626226 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:45.626198    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:45.786430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:45.786395    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:46.144729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:46.144685    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:46.522836 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:46.522796    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:46.917256 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:46.917170    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:47.329341 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:47.329299    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:47.749522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:47.746918    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:48.112055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:48.111972    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:48.237608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:48.237575    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:48.830008 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:48.829974    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:49.031192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:49.031161    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:49.717908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:49.717871    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:50.708113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:50.707939    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:51.890612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:51.885723    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:52.587958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:52.587926    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:53.358212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:53.358176    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:54.030343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:54.030305    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"]
Nov 03 20:35:54.126048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:54.126016    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:54.262414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:54.262381    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:54.641783 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:54.641736    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:54.777834 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:54.777798    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:55.437785 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:55.437742    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:55.578522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:55.576225    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:56.194649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:56.194501    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:56.851718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:56.851680    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:57.561643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:57.561604    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:58.246722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:58.246675    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:58.941097 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:58.941060    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:59.039506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:59.033561    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt" podUID="11bf2ea4-c852-4165-854c-ce2ad2018274" containerName="payload-processing" containerID="cri-o://bb3bccd01f7256713dfd8302ce1f99976de8c6976b1ef3a5497823db6b518b52" gracePeriod=25
Nov 03 20:35:59.373083 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:59.373053    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"
Nov 03 20:35:59.420513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:59.420456    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11bf2ea4-c852-4165-854c-ce2ad2018274-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11bf2ea4-c852-4165-854c-ce2ad2018274-plugins-config-volume\") pod \"11bf2ea4-c852-4165-854c-ce2ad2018274\" (UID: \"11bf2ea4-c852-4165-854c-ce2ad2018274\") "
Nov 03 20:35:59.420750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:59.420568    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11bf2ea4-c852-4165-854c-ce2ad2018274-kube-api-access-bjhww\" (UniqueName: \"kubernetes.io/projected/11bf2ea4-c852-4165-854c-ce2ad2018274-kube-api-access-bjhww\") pod \"11bf2ea4-c852-4165-854c-ce2ad2018274\" (UID: \"11bf2ea4-c852-4165-854c-ce2ad2018274\") "
Nov 03 20:35:59.420907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:59.420879    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11bf2ea4-c852-4165-854c-ce2ad2018274-plugins-config-volume" pod "11bf2ea4-c852-4165-854c-ce2ad2018274" (UID: "11bf2ea4-c852-4165-854c-ce2ad2018274"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:59.421279 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:59.421258    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11bf2ea4-c852-4165-854c-ce2ad2018274-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:59.423375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:59.423345    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11bf2ea4-c852-4165-854c-ce2ad2018274-kube-api-access-bjhww" pod "11bf2ea4-c852-4165-854c-ce2ad2018274" (UID: "11bf2ea4-c852-4165-854c-ce2ad2018274"). InnerVolumeSpecName "kube-api-access-bjhww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:59.522516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:59.522409    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjhww\" (UniqueName: \"kubernetes.io/projected/11bf2ea4-c852-4165-854c-ce2ad2018274-kube-api-access-bjhww\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:59.691506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:35:59.684298    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:00.098791 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:00.098750    5659 generic.go:356] "Generic (PLEG): container finished" podID="11bf2ea4-c852-4165-854c-ce2ad2018274" containerID="bb3bccd01f7256713dfd8302ce1f99976de8c6976b1ef3a5497823db6b518b52" exitCode=0
Nov 03 20:36:00.099306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:00.098802    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"
Nov 03 20:36:00.099306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:00.098844    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt" event={"ID":"11bf2ea4-c852-4165-854c-ce2ad2018274","Type":"ContainerDied","Data":"bb3bccd01f7256713dfd8302ce1f99976de8c6976b1ef3a5497823db6b518b52"}
Nov 03 20:36:00.099306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:00.098914    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt" event={"ID":"11bf2ea4-c852-4165-854c-ce2ad2018274","Type":"ContainerDied","Data":"25d3b4f69bd2a09ea1c07e4f8489e59742277c69f3af69bb47874c855f244464"}
Nov 03 20:36:00.099306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:00.098945    5659 scope.go:122] "RemoveContainer" containerID="bb3bccd01f7256713dfd8302ce1f99976de8c6976b1ef3a5497823db6b518b52"
Nov 03 20:36:00.110650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:00.110629    5659 scope.go:122] "RemoveContainer" containerID="bb3bccd01f7256713dfd8302ce1f99976de8c6976b1ef3a5497823db6b518b52"
Nov 03 20:36:00.110947 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:36:00.110927    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3bccd01f7256713dfd8302ce1f99976de8c6976b1ef3a5497823db6b518b52\": container with ID starting with bb3bccd01f7256713dfd8302ce1f99976de8c6976b1ef3a5497823db6b518b52 not found: ID does not exist" containerID="bb3bccd01f7256713dfd8302ce1f99976de8c6976b1ef3a5497823db6b518b52"
Nov 03 20:36:00.111004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:00.110957    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3bccd01f7256713dfd8302ce1f99976de8c6976b1ef3a5497823db6b518b52"} err="failed to get container status \"bb3bccd01f7256713dfd8302ce1f99976de8c6976b1ef3a5497823db6b518b52\": rpc error: code = NotFound desc = could not find container \"bb3bccd01f7256713dfd8302ce1f99976de8c6976b1ef3a5497823db6b518b52\": container with ID starting with bb3bccd01f7256713dfd8302ce1f99976de8c6976b1ef3a5497823db6b518b52 not found: ID does not exist"
Nov 03 20:36:00.127508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:00.127459    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"]
Nov 03 20:36:00.133962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:00.133935    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e1546-84f49b7cd8-qj9lt"]
Nov 03 20:36:00.471196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:00.462631    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:00.883394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:00.883308    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:01.156002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:01.151047    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:01.284103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:01.284069    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:01.841278 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:01.841244    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:01.972443 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:01.972396    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11bf2ea4-c852-4165-854c-ce2ad2018274" path="/var/lib/kubelet/pods/11bf2ea4-c852-4165-854c-ce2ad2018274/volumes"
Nov 03 20:36:02.524359 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:02.524323    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:03.169810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:03.169772    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"]
Nov 03 20:36:03.170330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:03.170297    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt" podUID="99bedd28-f7f9-4445-9bf5-56f3a82b1df8" containerName="maas-api" containerID="cri-o://abbcb0ec5a14fe2e557b400568277b1192128ef7317fb7a1a1e97a6bc61f8945" gracePeriod=30
Nov 03 20:36:03.214740 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:03.213646    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:03.288797 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:03.288758    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"]
Nov 03 20:36:03.289279 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:03.289241    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg" podUID="7349216a-564a-4fea-8cd8-4a8e8a6f3b2c" containerName="maas-api" containerID="cri-o://bab3ec68828d0de8a5a4fe773ce50cc40250bc44bddd469c9be3e892230825f6" gracePeriod=30
Nov 03 20:36:03.346505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:03.345884    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:03.643506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:03.640340    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:04.089259 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.085848    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:36:04.150519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.147565    5659 generic.go:356] "Generic (PLEG): container finished" podID="99bedd28-f7f9-4445-9bf5-56f3a82b1df8" containerID="abbcb0ec5a14fe2e557b400568277b1192128ef7317fb7a1a1e97a6bc61f8945" exitCode=0
Nov 03 20:36:04.150519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.147787    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt" event={"ID":"99bedd28-f7f9-4445-9bf5-56f3a82b1df8","Type":"ContainerDied","Data":"abbcb0ec5a14fe2e557b400568277b1192128ef7317fb7a1a1e97a6bc61f8945"}
Nov 03 20:36:04.150519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.147840    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt" event={"ID":"99bedd28-f7f9-4445-9bf5-56f3a82b1df8","Type":"ContainerDied","Data":"2c5507b8f2cb25af8f7883778fe6cfdfc9acb62cf71c2d63de21dd45ee213b36"}
Nov 03 20:36:04.150519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.147869    5659 scope.go:122] "RemoveContainer" containerID="abbcb0ec5a14fe2e557b400568277b1192128ef7317fb7a1a1e97a6bc61f8945"
Nov 03 20:36:04.150519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.147996    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"
Nov 03 20:36:04.155971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.155937    5659 generic.go:356] "Generic (PLEG): container finished" podID="7349216a-564a-4fea-8cd8-4a8e8a6f3b2c" containerID="bab3ec68828d0de8a5a4fe773ce50cc40250bc44bddd469c9be3e892230825f6" exitCode=0
Nov 03 20:36:04.156117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.155998    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg" event={"ID":"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c","Type":"ContainerDied","Data":"bab3ec68828d0de8a5a4fe773ce50cc40250bc44bddd469c9be3e892230825f6"}
Nov 03 20:36:04.181705 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.181673    5659 scope.go:122] "RemoveContainer" containerID="abbcb0ec5a14fe2e557b400568277b1192128ef7317fb7a1a1e97a6bc61f8945"
Nov 03 20:36:04.182685 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:36:04.182647    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbcb0ec5a14fe2e557b400568277b1192128ef7317fb7a1a1e97a6bc61f8945\": container with ID starting with abbcb0ec5a14fe2e557b400568277b1192128ef7317fb7a1a1e97a6bc61f8945 not found: ID does not exist" containerID="abbcb0ec5a14fe2e557b400568277b1192128ef7317fb7a1a1e97a6bc61f8945"
Nov 03 20:36:04.182868 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.182840    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbcb0ec5a14fe2e557b400568277b1192128ef7317fb7a1a1e97a6bc61f8945"} err="failed to get container status \"abbcb0ec5a14fe2e557b400568277b1192128ef7317fb7a1a1e97a6bc61f8945\": rpc error: code = NotFound desc = could not find container \"abbcb0ec5a14fe2e557b400568277b1192128ef7317fb7a1a1e97a6bc61f8945\": container with ID starting with abbcb0ec5a14fe2e557b400568277b1192128ef7317fb7a1a1e97a6bc61f8945 not found: ID does not exist"
Nov 03 20:36:04.217253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.217209    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:36:04.255076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.255023    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-maas-api-tls\") pod \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") "
Nov 03 20:36:04.255656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.255629    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-kube-api-access-pwnsc\" (UniqueName: \"kubernetes.io/projected/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-kube-api-access-pwnsc\") pod \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") "
Nov 03 20:36:04.255845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.255830    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-metrics-certs\") pod \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") "
Nov 03 20:36:04.256025 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.256007    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-openshift-service-ca\") pod \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\" (UID: \"99bedd28-f7f9-4445-9bf5-56f3a82b1df8\") "
Nov 03 20:36:04.257630 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.257605    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-openshift-service-ca" pod "99bedd28-f7f9-4445-9bf5-56f3a82b1df8" (UID: "99bedd28-f7f9-4445-9bf5-56f3a82b1df8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:04.295700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.295648    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-kube-api-access-pwnsc" pod "99bedd28-f7f9-4445-9bf5-56f3a82b1df8" (UID: "99bedd28-f7f9-4445-9bf5-56f3a82b1df8"). InnerVolumeSpecName "kube-api-access-pwnsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:04.295946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.295916    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-metrics-certs" pod "99bedd28-f7f9-4445-9bf5-56f3a82b1df8" (UID: "99bedd28-f7f9-4445-9bf5-56f3a82b1df8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:04.300538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.299881    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-maas-api-tls" pod "99bedd28-f7f9-4445-9bf5-56f3a82b1df8" (UID: "99bedd28-f7f9-4445-9bf5-56f3a82b1df8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:04.358260 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.358216    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-kube-api-access-dcgfw\" (UniqueName: \"kubernetes.io/projected/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-kube-api-access-dcgfw\") pod \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") "
Nov 03 20:36:04.358440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.358303    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-maas-api-tls\") pod \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") "
Nov 03 20:36:04.358511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.358475    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-metrics-certs\") pod \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") "
Nov 03 20:36:04.358570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.358556    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-openshift-service-ca\") pod \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\" (UID: \"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c\") "
Nov 03 20:36:04.358995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.358959    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-openshift-service-ca" pod "7349216a-564a-4fea-8cd8-4a8e8a6f3b2c" (UID: "7349216a-564a-4fea-8cd8-4a8e8a6f3b2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:04.360313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.360282    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:04.360433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.360319    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:04.360433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.360335    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:04.360433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.360350    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwnsc\" (UniqueName: \"kubernetes.io/projected/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-kube-api-access-pwnsc\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:04.360433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.360365    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bedd28-f7f9-4445-9bf5-56f3a82b1df8-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:04.371738 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.371700    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-metrics-certs" pod "7349216a-564a-4fea-8cd8-4a8e8a6f3b2c" (UID: "7349216a-564a-4fea-8cd8-4a8e8a6f3b2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:04.371960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.371753    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-kube-api-access-dcgfw" pod "7349216a-564a-4fea-8cd8-4a8e8a6f3b2c" (UID: "7349216a-564a-4fea-8cd8-4a8e8a6f3b2c"). InnerVolumeSpecName "kube-api-access-dcgfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:04.373461 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.373378    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-maas-api-tls" pod "7349216a-564a-4fea-8cd8-4a8e8a6f3b2c" (UID: "7349216a-564a-4fea-8cd8-4a8e8a6f3b2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:04.461442 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.461412    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:04.461442 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.461443    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcgfw\" (UniqueName: \"kubernetes.io/projected/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-kube-api-access-dcgfw\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:04.461655 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.461457    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:04.476596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.476565    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"]
Nov 03 20:36:04.482898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.482856    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt"]
Nov 03 20:36:04.537986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:04.537951    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:05.168934 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:05.168901    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"
Nov 03 20:36:05.168934 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:05.168912    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg" event={"ID":"7349216a-564a-4fea-8cd8-4a8e8a6f3b2c","Type":"ContainerDied","Data":"524d4db1b759a071ead39ddbebef3ef8d0ac891ef075db516e136de3e5ce50b0"}
Nov 03 20:36:05.169453 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:05.168983    5659 scope.go:122] "RemoveContainer" containerID="bab3ec68828d0de8a5a4fe773ce50cc40250bc44bddd469c9be3e892230825f6"
Nov 03 20:36:05.198538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:05.198460    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"]
Nov 03 20:36:05.202340 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:05.202313    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e1546-54b7f9876f-d7mqg"]
Nov 03 20:36:05.386575 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:05.386539    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:05.973468 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:05.973431    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7349216a-564a-4fea-8cd8-4a8e8a6f3b2c" path="/var/lib/kubelet/pods/7349216a-564a-4fea-8cd8-4a8e8a6f3b2c/volumes"
Nov 03 20:36:05.974093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:05.974063    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99bedd28-f7f9-4445-9bf5-56f3a82b1df8" path="/var/lib/kubelet/pods/99bedd28-f7f9-4445-9bf5-56f3a82b1df8/volumes"
Nov 03 20:36:06.144130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:06.144090    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:06.269781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:06.269654    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:06.457415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:06.457363    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-88a18e-c9fb8f8c-5dkpt" podUID="99bedd28-f7f9-4445-9bf5-56f3a82b1df8" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.102:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:36:06.881876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:06.881840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:07.435423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:07.435394    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:07.577315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:07.577263    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:07.722565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:07.722533    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:08.029762 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:08.029667    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:08.401530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:08.401417    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:09.041548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:09.041444    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:09.228553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:09.224251    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:09.835092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:09.835050    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:10.526731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:10.526699    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:10.665888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:10.665805    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:11.109951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:11.109899    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:11.258941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:11.258877    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:11.507492 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:11.507431    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:11.683554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:11.683515    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:11.959125 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:11.959091    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"]
Nov 03 20:36:11.959667 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:11.959634    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx" podUID="87a6ec71-2a2b-4e6c-aab0-56512872cb10" containerName="istio-proxy" containerID="cri-o://bcfc82f4b840049384bd666a9dbc6a89c28fcf4be94d55d6f55976ec189d32d4" gracePeriod=30
Nov 03 20:36:12.019201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:12.019163    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:12.418557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:12.410990    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:13.121822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:13.121785    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:13.830538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:13.830472    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:14.621751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:14.621714    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:15.219542 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:36:15.219470    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:36:15.397509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:15.388232    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:16.016322 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:16.016293    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:16.202576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:16.202534    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:16.330876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:16.330789    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:16.550767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:16.550729    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:17.106527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.106454    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:17.362048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.362020    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:36:17.376650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.376597    5659 generic.go:356] "Generic (PLEG): container finished" podID="87a6ec71-2a2b-4e6c-aab0-56512872cb10" containerID="bcfc82f4b840049384bd666a9dbc6a89c28fcf4be94d55d6f55976ec189d32d4" exitCode=0
Nov 03 20:36:17.376650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.376647    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"
Nov 03 20:36:17.376901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.376681    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx" event={"ID":"87a6ec71-2a2b-4e6c-aab0-56512872cb10","Type":"ContainerDied","Data":"bcfc82f4b840049384bd666a9dbc6a89c28fcf4be94d55d6f55976ec189d32d4"}
Nov 03 20:36:17.376901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.376748    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx" event={"ID":"87a6ec71-2a2b-4e6c-aab0-56512872cb10","Type":"ContainerDied","Data":"069bf541c028c57a8889938e538fa8ba37223918e830a1bfba9099a803b7c61a"}
Nov 03 20:36:17.376901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.376780    5659 scope.go:122] "RemoveContainer" containerID="bcfc82f4b840049384bd666a9dbc6a89c28fcf4be94d55d6f55976ec189d32d4"
Nov 03 20:36:17.395825 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.395743    5659 scope.go:122] "RemoveContainer" containerID="bcfc82f4b840049384bd666a9dbc6a89c28fcf4be94d55d6f55976ec189d32d4"
Nov 03 20:36:17.396853 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:36:17.396825    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcfc82f4b840049384bd666a9dbc6a89c28fcf4be94d55d6f55976ec189d32d4\": container with ID starting with bcfc82f4b840049384bd666a9dbc6a89c28fcf4be94d55d6f55976ec189d32d4 not found: ID does not exist" containerID="bcfc82f4b840049384bd666a9dbc6a89c28fcf4be94d55d6f55976ec189d32d4"
Nov 03 20:36:17.396996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.396863    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcfc82f4b840049384bd666a9dbc6a89c28fcf4be94d55d6f55976ec189d32d4"} err="failed to get container status \"bcfc82f4b840049384bd666a9dbc6a89c28fcf4be94d55d6f55976ec189d32d4\": rpc error: code = NotFound desc = could not find container \"bcfc82f4b840049384bd666a9dbc6a89c28fcf4be94d55d6f55976ec189d32d4\": container with ID starting with bcfc82f4b840049384bd666a9dbc6a89c28fcf4be94d55d6f55976ec189d32d4 not found: ID does not exist"
Nov 03 20:36:17.471538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.471473    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-certs\") pod \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") "
Nov 03 20:36:17.471733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.471576    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istiod-ca-cert\") pod \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") "
Nov 03 20:36:17.471733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.471697    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-kube-api-access-85qwm\" (UniqueName: \"kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-kube-api-access-85qwm\") pod \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") "
Nov 03 20:36:17.471827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.471776    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-envoy\") pod \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") "
Nov 03 20:36:17.471863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.471842    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-credential-socket\") pod \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") "
Nov 03 20:36:17.471863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.471840    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-certs" pod "87a6ec71-2a2b-4e6c-aab0-56512872cb10" (UID: "87a6ec71-2a2b-4e6c-aab0-56512872cb10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:17.471937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.471889    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-podinfo\") pod \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") "
Nov 03 20:36:17.471973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.471933    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-socket\") pod \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") "
Nov 03 20:36:17.472009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.471985    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-data\") pod \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") "
Nov 03 20:36:17.472060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.472038    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-token\" (UniqueName: \"kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-token\") pod \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\" (UID: \"87a6ec71-2a2b-4e6c-aab0-56512872cb10\") "
Nov 03 20:36:17.472686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.472036    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istiod-ca-cert" pod "87a6ec71-2a2b-4e6c-aab0-56512872cb10" (UID: "87a6ec71-2a2b-4e6c-aab0-56512872cb10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:17.473429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.473365    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-data" pod "87a6ec71-2a2b-4e6c-aab0-56512872cb10" (UID: "87a6ec71-2a2b-4e6c-aab0-56512872cb10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:17.473847 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.473825    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-credential-socket" pod "87a6ec71-2a2b-4e6c-aab0-56512872cb10" (UID: "87a6ec71-2a2b-4e6c-aab0-56512872cb10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:17.474200 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.474180    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:17.474332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.474316    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:17.474462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.474416    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:17.474687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.474668    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:17.474846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.474566    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-podinfo" pod "87a6ec71-2a2b-4e6c-aab0-56512872cb10" (UID: "87a6ec71-2a2b-4e6c-aab0-56512872cb10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:36:17.474957 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.474940    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-kube-api-access-85qwm" pod "87a6ec71-2a2b-4e6c-aab0-56512872cb10" (UID: "87a6ec71-2a2b-4e6c-aab0-56512872cb10"). InnerVolumeSpecName "kube-api-access-85qwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:17.475309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.475282    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-token" pod "87a6ec71-2a2b-4e6c-aab0-56512872cb10" (UID: "87a6ec71-2a2b-4e6c-aab0-56512872cb10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:17.475444 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.475424    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-socket" pod "87a6ec71-2a2b-4e6c-aab0-56512872cb10" (UID: "87a6ec71-2a2b-4e6c-aab0-56512872cb10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:17.475560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.475444    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-envoy" pod "87a6ec71-2a2b-4e6c-aab0-56512872cb10" (UID: "87a6ec71-2a2b-4e6c-aab0-56512872cb10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:17.577283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.577188    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:17.577283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.577234    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:17.577283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.577249    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:17.577283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.577264    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85qwm\" (UniqueName: \"kubernetes.io/projected/87a6ec71-2a2b-4e6c-aab0-56512872cb10-kube-api-access-85qwm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:17.577283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.577279    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a6ec71-2a2b-4e6c-aab0-56512872cb10-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:17.708427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.708390    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"]
Nov 03 20:36:17.715513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.715470    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-88a18e-openshift-default-7687cc86c-wtkzx"]
Nov 03 20:36:17.928066 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.927976    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:17.986906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:17.986871    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a6ec71-2a2b-4e6c-aab0-56512872cb10" path="/var/lib/kubelet/pods/87a6ec71-2a2b-4e6c-aab0-56512872cb10/volumes"
Nov 03 20:36:18.632949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:18.632914    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:19.263286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:19.263244    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:19.748840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:19.748796    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:20.631744 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:20.631707    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:21.363537 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:21.363425    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:22.053516 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:22.052573    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:22.478678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:22.478637    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:22.867587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:22.867469    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:23.103767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:23.103737    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:23.247443 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:23.247379    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:23.691872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:23.691783    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:23.892774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:23.892722    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:24.125048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:24.125009    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:24.640435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:24.640400    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:24.868073 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:24.868037    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:25.350609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:25.349277    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:26.062415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:26.062378    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:26.760221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:26.760184    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:27.414701 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:27.414670    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"]
Nov 03 20:36:27.469974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:27.469937    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:27.599631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:27.599595    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:28.067123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:28.067084    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:28.366000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:28.365912    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:29.058875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:29.058842    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:29.758567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:29.758515    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:30.771450 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:30.771397    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:31.599961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:31.599922    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:31.964382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:31.964343    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:32.116944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:32.116905    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:32.617219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:32.617179    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:32.845969 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:32.845938    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:33.221339 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:33.221303    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:33.375056 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:33.375019    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:33.731095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:33.731057    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:34.621464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:34.621414    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:35.221648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:35.221612    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:35.420533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:35.420499    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:36.187870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:36.187833    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:36.828217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:36.828181    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:37.567271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:37.567189    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:38.364520 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:38.364466    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:39.165000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:39.163075    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:40.462269 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:40.462229    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:41.073504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:41.061830    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:41.238105 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:41.238068    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:41.741429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:41.741366    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:42.441355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.441313    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:42.464316 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.464126    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main" containerID="cri-o://1466b9e49f1926675a97766a0ee53a5424b93d341d38bbb1383c74fbe34dc2be" gracePeriod=45
Nov 03 20:36:42.766260 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.766220    5659 generic.go:356] "Generic (PLEG): container finished" podID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerID="1466b9e49f1926675a97766a0ee53a5424b93d341d38bbb1383c74fbe34dc2be" exitCode=0
Nov 03 20:36:42.766637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.766309    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" event={"ID":"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85","Type":"ContainerDied","Data":"1466b9e49f1926675a97766a0ee53a5424b93d341d38bbb1383c74fbe34dc2be"}
Nov 03 20:36:42.766637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.766369    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws" event={"ID":"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85","Type":"ContainerDied","Data":"9ddf2288a47411900936d1b58fdbeff56ef574bc35fb55bea75fd6258089c4ba"}
Nov 03 20:36:42.766637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.766387    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ddf2288a47411900936d1b58fdbeff56ef574bc35fb55bea75fd6258089c4ba"
Nov 03 20:36:42.774958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.774927    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:36:42.904920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.904870    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tmp-dir\") pod \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") "
Nov 03 20:36:42.905119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.904934    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-kube-api-access-xzqhm\" (UniqueName: \"kubernetes.io/projected/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-kube-api-access-xzqhm\") pod \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") "
Nov 03 20:36:42.905119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.904972    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-home\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-home\") pod \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") "
Nov 03 20:36:42.905119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.905004    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-model-cache\") pod \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") "
Nov 03 20:36:42.905315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.905197    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tmp-dir" pod "b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" (UID: "b309ad2b-6391-4e09-8bbc-ae2ca1bdef85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.905315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.905233    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-dshm\") pod \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") "
Nov 03 20:36:42.905436 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.905360    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tls-certs\" (UniqueName: \"kubernetes.io/secret/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tls-certs\") pod \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\" (UID: \"b309ad2b-6391-4e09-8bbc-ae2ca1bdef85\") "
Nov 03 20:36:42.905547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.905451    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-home" pod "b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" (UID: "b309ad2b-6391-4e09-8bbc-ae2ca1bdef85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.906137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.906063    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.906137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.906089    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.906592 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.906552    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-model-cache" pod "b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" (UID: "b309ad2b-6391-4e09-8bbc-ae2ca1bdef85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.911669 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.911637    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-kube-api-access-xzqhm" pod "b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" (UID: "b309ad2b-6391-4e09-8bbc-ae2ca1bdef85"). InnerVolumeSpecName "kube-api-access-xzqhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:42.912166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.912142    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-dshm" pod "b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" (UID: "b309ad2b-6391-4e09-8bbc-ae2ca1bdef85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.932075 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:42.929766    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tls-certs" pod "b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" (UID: "b309ad2b-6391-4e09-8bbc-ae2ca1bdef85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:43.032088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:43.031646    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:43.032088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:43.031681    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:43.032088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:43.031697    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzqhm\" (UniqueName: \"kubernetes.io/projected/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-kube-api-access-xzqhm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:43.032088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:43.031715    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:43.161285 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:43.161247    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:43.773843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:43.773802    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"
Nov 03 20:36:43.799721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:43.799691    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"]
Nov 03 20:36:43.804341 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:43.804313    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/premium-w4-3fbb1a-kserve-c4dd4dbcf-8s5ws"]
Nov 03 20:36:43.999506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:43.982597    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:44.016082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:44.016045    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" path="/var/lib/kubelet/pods/b309ad2b-6391-4e09-8bbc-ae2ca1bdef85/volumes"
Nov 03 20:36:44.702504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:44.699274    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:45.219545 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:36:45.219509    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:36:45.334003 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:45.333967    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:46.023037 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:46.022998    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:46.281820 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:46.281734    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:46.414700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:46.414660    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:46.866241 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:46.866205    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:47.001565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:47.000344    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:47.186298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:47.186209    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"]
Nov 03 20:36:47.186524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:47.186478    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" podUID="b2fe4f76-2186-4315-bb74-db93367de753" containerName="istio-proxy" containerID="cri-o://0b8e0b92a00848ffd1d93702dfe8474f28204a3e5db8cdb5baeadc767a7d946a" gracePeriod=30
Nov 03 20:36:47.254069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:47.254030    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:47.688875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:47.688840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:47.854904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:47.854869    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:47.981738 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:47.981703    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:48.550194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:48.550156    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:49.286108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:49.286076    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:49.943678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:49.943644    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:50.765425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:50.765316    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:51.179534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:51.179426    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:36:51.179696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:51.179549    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" podUID="b2fe4f76-2186-4315-bb74-db93367de753" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:36:51.488840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:51.488802    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:52.217362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:52.217328    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:52.640927 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:52.640833    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"]
Nov 03 20:36:52.646301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:52.646268    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"]
Nov 03 20:36:52.661377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:52.661346    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"]
Nov 03 20:36:52.782566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:52.782530    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:52.867292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:52.867222    5659 generic.go:356] "Generic (PLEG): container finished" podID="b2fe4f76-2186-4315-bb74-db93367de753" containerID="0b8e0b92a00848ffd1d93702dfe8474f28204a3e5db8cdb5baeadc767a7d946a" exitCode=0
Nov 03 20:36:52.867524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:52.867464    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" event={"ID":"b2fe4f76-2186-4315-bb74-db93367de753","Type":"ContainerDied","Data":"0b8e0b92a00848ffd1d93702dfe8474f28204a3e5db8cdb5baeadc767a7d946a"}
Nov 03 20:36:52.942038 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:52.941984    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:36:52.994565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:52.994526    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:53.055939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.050588    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-kube-api-access-d672n\" (UniqueName: \"kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-kube-api-access-d672n\") pod \"b2fe4f76-2186-4315-bb74-db93367de753\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") "
Nov 03 20:36:53.055939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.050686    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2fe4f76-2186-4315-bb74-db93367de753-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fe4f76-2186-4315-bb74-db93367de753-istiod-ca-cert\") pod \"b2fe4f76-2186-4315-bb74-db93367de753\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") "
Nov 03 20:36:53.055939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.050769    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-envoy\") pod \"b2fe4f76-2186-4315-bb74-db93367de753\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") "
Nov 03 20:36:53.055939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.050831    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-istio-token\" (UniqueName: \"kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-istio-token\") pod \"b2fe4f76-2186-4315-bb74-db93367de753\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") "
Nov 03 20:36:53.055939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.050877    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-data\") pod \"b2fe4f76-2186-4315-bb74-db93367de753\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") "
Nov 03 20:36:53.055939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.050914    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-credential-socket\") pod \"b2fe4f76-2186-4315-bb74-db93367de753\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") "
Nov 03 20:36:53.055939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.051034    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-certs\") pod \"b2fe4f76-2186-4315-bb74-db93367de753\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") "
Nov 03 20:36:53.055939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.051107    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-socket\") pod \"b2fe4f76-2186-4315-bb74-db93367de753\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") "
Nov 03 20:36:53.055939 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.051208    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2fe4f76-2186-4315-bb74-db93367de753-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fe4f76-2186-4315-bb74-db93367de753-istio-podinfo\") pod \"b2fe4f76-2186-4315-bb74-db93367de753\" (UID: \"b2fe4f76-2186-4315-bb74-db93367de753\") "
Nov 03 20:36:53.066416 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.066377    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-socket" pod "b2fe4f76-2186-4315-bb74-db93367de753" (UID: "b2fe4f76-2186-4315-bb74-db93367de753"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:53.066615 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.066575    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-credential-socket" pod "b2fe4f76-2186-4315-bb74-db93367de753" (UID: "b2fe4f76-2186-4315-bb74-db93367de753"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:53.066737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.066722    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-certs" pod "b2fe4f76-2186-4315-bb74-db93367de753" (UID: "b2fe4f76-2186-4315-bb74-db93367de753"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:53.067057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.067006    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-data" pod "b2fe4f76-2186-4315-bb74-db93367de753" (UID: "b2fe4f76-2186-4315-bb74-db93367de753"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:53.067764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.067736    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2fe4f76-2186-4315-bb74-db93367de753-istiod-ca-cert" pod "b2fe4f76-2186-4315-bb74-db93367de753" (UID: "b2fe4f76-2186-4315-bb74-db93367de753"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:53.092200 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.092164    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2fe4f76-2186-4315-bb74-db93367de753-istio-podinfo" pod "b2fe4f76-2186-4315-bb74-db93367de753" (UID: "b2fe4f76-2186-4315-bb74-db93367de753"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:36:53.099322 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.097807    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-istio-token" pod "b2fe4f76-2186-4315-bb74-db93367de753" (UID: "b2fe4f76-2186-4315-bb74-db93367de753"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:53.099322 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.097936    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-envoy" pod "b2fe4f76-2186-4315-bb74-db93367de753" (UID: "b2fe4f76-2186-4315-bb74-db93367de753"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:53.101970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.101871    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-kube-api-access-d672n" pod "b2fe4f76-2186-4315-bb74-db93367de753" (UID: "b2fe4f76-2186-4315-bb74-db93367de753"). InnerVolumeSpecName "kube-api-access-d672n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:53.158543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.158448    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:53.158543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.158499    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:53.158543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.158509    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fe4f76-2186-4315-bb74-db93367de753-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:53.158543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.158542    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d672n\" (UniqueName: \"kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-kube-api-access-d672n\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:53.158813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.158553    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fe4f76-2186-4315-bb74-db93367de753-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:53.158813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.158562    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:53.158813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.158571    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fe4f76-2186-4315-bb74-db93367de753-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:53.158813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.158579    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:53.158813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.158587    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fe4f76-2186-4315-bb74-db93367de753-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:53.274095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.274057    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:53.622685 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.622655    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:53.878410 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.878317    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"
Nov 03 20:36:53.878592 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.878309    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx" event={"ID":"b2fe4f76-2186-4315-bb74-db93367de753","Type":"ContainerDied","Data":"e579dabca1ac13a5fae6134bfa38f5e522e9eac3f8b76b2f4fb85840bfbc9322"}
Nov 03 20:36:53.878592 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.878557    5659 scope.go:122] "RemoveContainer" containerID="0b8e0b92a00848ffd1d93702dfe8474f28204a3e5db8cdb5baeadc767a7d946a"
Nov 03 20:36:53.909517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.909470    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"]
Nov 03 20:36:53.918412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.918380    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-81a5ae-openshift-default-59cb685c59-xvgzx"]
Nov 03 20:36:53.974286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:53.974250    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2fe4f76-2186-4315-bb74-db93367de753" path="/var/lib/kubelet/pods/b2fe4f76-2186-4315-bb74-db93367de753/volumes"
Nov 03 20:36:54.331046 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:54.331010    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:54.980401 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:54.980363    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:55.660555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:55.660518    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:56.350503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:56.347181    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:56.844505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:56.840729    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"]
Nov 03 20:36:56.844505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:56.840986    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s" podUID="2f904191-5cd6-4550-ad9d-431ba37472cd" containerName="maas-api" containerID="cri-o://8144380d26806e17efdf860f95a2ffe2fe016b660f58afd894b96f4bbed25b72" gracePeriod=30
Nov 03 20:36:56.895119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:56.895017    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:57.211373 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.211337    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:57.592572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.592537    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 20:36:57.675803 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.675763    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-metrics-certs\") pod \"2f904191-5cd6-4550-ad9d-431ba37472cd\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") "
Nov 03 20:36:57.675986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.675817    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f904191-5cd6-4550-ad9d-431ba37472cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f904191-5cd6-4550-ad9d-431ba37472cd-openshift-service-ca\") pod \"2f904191-5cd6-4550-ad9d-431ba37472cd\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") "
Nov 03 20:36:57.675986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.675889    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f904191-5cd6-4550-ad9d-431ba37472cd-kube-api-access-ssxv2\" (UniqueName: \"kubernetes.io/projected/2f904191-5cd6-4550-ad9d-431ba37472cd-kube-api-access-ssxv2\") pod \"2f904191-5cd6-4550-ad9d-431ba37472cd\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") "
Nov 03 20:36:57.676061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.676010    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-maas-api-tls\") pod \"2f904191-5cd6-4550-ad9d-431ba37472cd\" (UID: \"2f904191-5cd6-4550-ad9d-431ba37472cd\") "
Nov 03 20:36:57.676661 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.676630    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f904191-5cd6-4550-ad9d-431ba37472cd-openshift-service-ca" pod "2f904191-5cd6-4550-ad9d-431ba37472cd" (UID: "2f904191-5cd6-4550-ad9d-431ba37472cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:57.678941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.678911    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-maas-api-tls" pod "2f904191-5cd6-4550-ad9d-431ba37472cd" (UID: "2f904191-5cd6-4550-ad9d-431ba37472cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:57.679676 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.679640    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f904191-5cd6-4550-ad9d-431ba37472cd-kube-api-access-ssxv2" pod "2f904191-5cd6-4550-ad9d-431ba37472cd" (UID: "2f904191-5cd6-4550-ad9d-431ba37472cd"). InnerVolumeSpecName "kube-api-access-ssxv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:57.679818 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.679712    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-metrics-certs" pod "2f904191-5cd6-4550-ad9d-431ba37472cd" (UID: "2f904191-5cd6-4550-ad9d-431ba37472cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:57.778937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.778839    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssxv2\" (UniqueName: \"kubernetes.io/projected/2f904191-5cd6-4550-ad9d-431ba37472cd-kube-api-access-ssxv2\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:57.778937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.778887    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:57.778937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.778902    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f904191-5cd6-4550-ad9d-431ba37472cd-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:57.778937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.778916    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f904191-5cd6-4550-ad9d-431ba37472cd-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:57.923203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.923168    5659 generic.go:356] "Generic (PLEG): container finished" podID="2f904191-5cd6-4550-ad9d-431ba37472cd" containerID="8144380d26806e17efdf860f95a2ffe2fe016b660f58afd894b96f4bbed25b72" exitCode=0
Nov 03 20:36:57.923721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.923218    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"
Nov 03 20:36:57.923721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.923258    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s" event={"ID":"2f904191-5cd6-4550-ad9d-431ba37472cd","Type":"ContainerDied","Data":"8144380d26806e17efdf860f95a2ffe2fe016b660f58afd894b96f4bbed25b72"}
Nov 03 20:36:57.923721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.923328    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s" event={"ID":"2f904191-5cd6-4550-ad9d-431ba37472cd","Type":"ContainerDied","Data":"0ee064ba02840d5100ff8042cf611209e3ce90a7a695144c319867393042c41f"}
Nov 03 20:36:57.923721 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.923359    5659 scope.go:122] "RemoveContainer" containerID="8144380d26806e17efdf860f95a2ffe2fe016b660f58afd894b96f4bbed25b72"
Nov 03 20:36:57.995501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:57.995042    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:58.019505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:58.016811    5659 scope.go:122] "RemoveContainer" containerID="8144380d26806e17efdf860f95a2ffe2fe016b660f58afd894b96f4bbed25b72"
Nov 03 20:36:58.026700 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:36:58.026663    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8144380d26806e17efdf860f95a2ffe2fe016b660f58afd894b96f4bbed25b72\": container with ID starting with 8144380d26806e17efdf860f95a2ffe2fe016b660f58afd894b96f4bbed25b72 not found: ID does not exist" containerID="8144380d26806e17efdf860f95a2ffe2fe016b660f58afd894b96f4bbed25b72"
Nov 03 20:36:58.026982 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:58.026932    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8144380d26806e17efdf860f95a2ffe2fe016b660f58afd894b96f4bbed25b72"} err="failed to get container status \"8144380d26806e17efdf860f95a2ffe2fe016b660f58afd894b96f4bbed25b72\": rpc error: code = NotFound desc = could not find container \"8144380d26806e17efdf860f95a2ffe2fe016b660f58afd894b96f4bbed25b72\": container with ID starting with 8144380d26806e17efdf860f95a2ffe2fe016b660f58afd894b96f4bbed25b72 not found: ID does not exist"
Nov 03 20:36:58.029731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:58.029602    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"]
Nov 03 20:36:58.029731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:58.029646    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3fbb1a-b8b578bfc-w7b8s"]
Nov 03 20:36:58.665509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:58.661300    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:59.443693 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:59.443658    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:59.900930 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:59.900887    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:59.975149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:36:59.975110    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f904191-5cd6-4550-ad9d-431ba37472cd" path="/var/lib/kubelet/pods/2f904191-5cd6-4550-ad9d-431ba37472cd/volumes"
Nov 03 20:37:00.033535 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:00.033458    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:00.299633 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:00.299602    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:00.511020 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:00.510929    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:01.233693 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:01.233658    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:01.976816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:01.976706    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:02.886580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:02.868781    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:03.750685 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:03.747739    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:04.820503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:04.610813    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:04.871508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:04.859122    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:05.605554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:05.604063    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:05.913566 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:05.909132    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:06.063469 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:06.063425    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:06.313687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:06.313622    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:06.511538 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:06.511115    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:06.711610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:06.701779    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:06.868925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:06.863006    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"]
Nov 03 20:37:06.868925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:06.863315    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerName="istio-proxy" containerID="cri-o://c37184347e131df697cb225a8947a058c382cbf3e698a1c6ea07d302bdecffad" gracePeriod=30
Nov 03 20:37:06.868925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:06.863734    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"]
Nov 03 20:37:06.868925 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:06.863948    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc" podUID="f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" containerName="istio-proxy" containerID="cri-o://f5b8806ce4d79e7f1e325fcfded956cc4cae2a98d1012022f1cb9d0dc6b907fa" gracePeriod=30
Nov 03 20:37:06.888705 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:06.885451    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:07.359059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:07.358988    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:07.591194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:07.591157    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:07.670356 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:07.670245    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main" containerID="cri-o://a3607cb980e7c9ebf6ba92135b100a6fcc54ea0c0cc7b734dc9c22e758b71af5" gracePeriod=45
Nov 03 20:37:07.671668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:07.671633    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" containerID="cri-o://7aafe1e6315d850f3027f8ec212e633a96e2410bbefbfd3754202e3dab11587a" gracePeriod=45
Nov 03 20:37:07.694822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:07.694788    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" containerID="cri-o://d8274317608e774f10d27cc611a9c4aec293fc5e7eb16ed7e3e0fe20c3832878" gracePeriod=45
Nov 03 20:37:07.802924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:07.802881    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": dial tcp 10.133.0.47:8000: connect: connection refused"
Nov 03 20:37:07.903823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:07.903781    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": dial tcp 10.133.0.48:8000: connect: connection refused"
Nov 03 20:37:08.026122 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.026088    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 20:37:08.036731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.036688    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 20:37:08.039044 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.039022    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 20:37:08.091218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.091178    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tls-certs\") pod \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") "
Nov 03 20:37:08.091419 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.091255    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c530754-24a3-4aa7-99cc-99bde27283b3-kube-api-access-lfs6c\" (UniqueName: \"kubernetes.io/projected/0c530754-24a3-4aa7-99cc-99bde27283b3-kube-api-access-lfs6c\") pod \"0c530754-24a3-4aa7-99cc-99bde27283b3\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") "
Nov 03 20:37:08.091419 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.091340    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7205a658-0441-4700-a96e-8323bf72b395-kube-api-access-6rjbg\" (UniqueName: \"kubernetes.io/projected/7205a658-0441-4700-a96e-8323bf72b395-kube-api-access-6rjbg\") pod \"7205a658-0441-4700-a96e-8323bf72b395\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") "
Nov 03 20:37:08.091419 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.091388    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-home\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-home\") pod \"7205a658-0441-4700-a96e-8323bf72b395\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") "
Nov 03 20:37:08.091638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.091434    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-model-cache\") pod \"0c530754-24a3-4aa7-99cc-99bde27283b3\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") "
Nov 03 20:37:08.091638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.091595    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-tmp-dir\") pod \"0c530754-24a3-4aa7-99cc-99bde27283b3\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") "
Nov 03 20:37:08.091742 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.091687    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-dshm\") pod \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") "
Nov 03 20:37:08.091791 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.091766    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-dshm\") pod \"0c530754-24a3-4aa7-99cc-99bde27283b3\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") "
Nov 03 20:37:08.091841 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.091801    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-model-cache\") pod \"7205a658-0441-4700-a96e-8323bf72b395\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") "
Nov 03 20:37:08.091949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.091868    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-dshm\") pod \"7205a658-0441-4700-a96e-8323bf72b395\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") "
Nov 03 20:37:08.092055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.091990    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tmp-dir\") pod \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") "
Nov 03 20:37:08.092055 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.092037    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-home\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-home\") pod \"0c530754-24a3-4aa7-99cc-99bde27283b3\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") "
Nov 03 20:37:08.092170 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.092107    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-kube-api-access-66h52\" (UniqueName: \"kubernetes.io/projected/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-kube-api-access-66h52\") pod \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") "
Nov 03 20:37:08.092280 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.092259    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-model-cache\") pod \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") "
Nov 03 20:37:08.092415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.092399    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-tmp-dir\") pod \"7205a658-0441-4700-a96e-8323bf72b395\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") "
Nov 03 20:37:08.092475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.092462    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7205a658-0441-4700-a96e-8323bf72b395-tls-certs\" (UniqueName: \"kubernetes.io/secret/7205a658-0441-4700-a96e-8323bf72b395-tls-certs\") pod \"7205a658-0441-4700-a96e-8323bf72b395\" (UID: \"7205a658-0441-4700-a96e-8323bf72b395\") "
Nov 03 20:37:08.092475 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.092501    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-home\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-home\") pod \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\" (UID: \"6e5e3878-4eae-4f9e-8ae4-9483719cfbda\") "
Nov 03 20:37:08.092631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.092545    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c530754-24a3-4aa7-99cc-99bde27283b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c530754-24a3-4aa7-99cc-99bde27283b3-tls-certs\") pod \"0c530754-24a3-4aa7-99cc-99bde27283b3\" (UID: \"0c530754-24a3-4aa7-99cc-99bde27283b3\") "
Nov 03 20:37:08.097060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.096850    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-tmp-dir" pod "7205a658-0441-4700-a96e-8323bf72b395" (UID: "7205a658-0441-4700-a96e-8323bf72b395"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.099024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.097203    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-home" pod "6e5e3878-4eae-4f9e-8ae4-9483719cfbda" (UID: "6e5e3878-4eae-4f9e-8ae4-9483719cfbda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.099024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.097248    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tmp-dir" pod "6e5e3878-4eae-4f9e-8ae4-9483719cfbda" (UID: "6e5e3878-4eae-4f9e-8ae4-9483719cfbda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.099024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.097507    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-home" pod "0c530754-24a3-4aa7-99cc-99bde27283b3" (UID: "0c530754-24a3-4aa7-99cc-99bde27283b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.099024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.097525    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-model-cache" pod "0c530754-24a3-4aa7-99cc-99bde27283b3" (UID: "0c530754-24a3-4aa7-99cc-99bde27283b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.099024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.097863    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-home" pod "7205a658-0441-4700-a96e-8323bf72b395" (UID: "7205a658-0441-4700-a96e-8323bf72b395"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.099024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.098086    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-tmp-dir" pod "0c530754-24a3-4aa7-99cc-99bde27283b3" (UID: "0c530754-24a3-4aa7-99cc-99bde27283b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.099024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.098288    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-model-cache" pod "6e5e3878-4eae-4f9e-8ae4-9483719cfbda" (UID: "6e5e3878-4eae-4f9e-8ae4-9483719cfbda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.101067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.100799    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tls-certs" pod "6e5e3878-4eae-4f9e-8ae4-9483719cfbda" (UID: "6e5e3878-4eae-4f9e-8ae4-9483719cfbda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:08.102494 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.102434    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-dshm" pod "7205a658-0441-4700-a96e-8323bf72b395" (UID: "7205a658-0441-4700-a96e-8323bf72b395"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.102621 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.102601    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-dshm" pod "0c530754-24a3-4aa7-99cc-99bde27283b3" (UID: "0c530754-24a3-4aa7-99cc-99bde27283b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.102703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.102678    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-dshm" pod "6e5e3878-4eae-4f9e-8ae4-9483719cfbda" (UID: "6e5e3878-4eae-4f9e-8ae4-9483719cfbda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.102773 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.102696    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-model-cache" pod "7205a658-0441-4700-a96e-8323bf72b395" (UID: "7205a658-0441-4700-a96e-8323bf72b395"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.102837 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.102817    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7205a658-0441-4700-a96e-8323bf72b395-kube-api-access-6rjbg" pod "7205a658-0441-4700-a96e-8323bf72b395" (UID: "7205a658-0441-4700-a96e-8323bf72b395"). InnerVolumeSpecName "kube-api-access-6rjbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:08.102903 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.102844    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-kube-api-access-66h52" pod "6e5e3878-4eae-4f9e-8ae4-9483719cfbda" (UID: "6e5e3878-4eae-4f9e-8ae4-9483719cfbda"). InnerVolumeSpecName "kube-api-access-66h52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:08.103167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.103145    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c530754-24a3-4aa7-99cc-99bde27283b3-kube-api-access-lfs6c" pod "0c530754-24a3-4aa7-99cc-99bde27283b3" (UID: "0c530754-24a3-4aa7-99cc-99bde27283b3"). InnerVolumeSpecName "kube-api-access-lfs6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:08.104106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.104078    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c530754-24a3-4aa7-99cc-99bde27283b3-tls-certs" pod "0c530754-24a3-4aa7-99cc-99bde27283b3" (UID: "0c530754-24a3-4aa7-99cc-99bde27283b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:08.104271 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.104251    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7205a658-0441-4700-a96e-8323bf72b395-tls-certs" pod "7205a658-0441-4700-a96e-8323bf72b395" (UID: "7205a658-0441-4700-a96e-8323bf72b395"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:08.161018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.160979    5659 generic.go:356] "Generic (PLEG): container finished" podID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerID="7aafe1e6315d850f3027f8ec212e633a96e2410bbefbfd3754202e3dab11587a" exitCode=0
Nov 03 20:37:08.161218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.161123    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" event={"ID":"0c530754-24a3-4aa7-99cc-99bde27283b3","Type":"ContainerDied","Data":"7aafe1e6315d850f3027f8ec212e633a96e2410bbefbfd3754202e3dab11587a"}
Nov 03 20:37:08.161218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.161158    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m" event={"ID":"0c530754-24a3-4aa7-99cc-99bde27283b3","Type":"ContainerDied","Data":"c69cbbdb56294d176578ce112ec85f65d216aaf78a452ef07a5c47f09baaa9bf"}
Nov 03 20:37:08.161218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.161180    5659 scope.go:122] "RemoveContainer" containerID="7aafe1e6315d850f3027f8ec212e633a96e2410bbefbfd3754202e3dab11587a"
Nov 03 20:37:08.161417 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.161334    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"
Nov 03 20:37:08.170219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.167964    5659 generic.go:356] "Generic (PLEG): container finished" podID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerID="a3607cb980e7c9ebf6ba92135b100a6fcc54ea0c0cc7b734dc9c22e758b71af5" exitCode=0
Nov 03 20:37:08.170219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.168031    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"
Nov 03 20:37:08.170219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.168152    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" event={"ID":"6e5e3878-4eae-4f9e-8ae4-9483719cfbda","Type":"ContainerDied","Data":"a3607cb980e7c9ebf6ba92135b100a6fcc54ea0c0cc7b734dc9c22e758b71af5"}
Nov 03 20:37:08.170219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.168187    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk" event={"ID":"6e5e3878-4eae-4f9e-8ae4-9483719cfbda","Type":"ContainerDied","Data":"930bfd25721f6cf7446e4120d5d55fe0a362cf8eef45fe389e27778d9a979b03"}
Nov 03 20:37:08.170578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.170227    5659 generic.go:356] "Generic (PLEG): container finished" podID="7205a658-0441-4700-a96e-8323bf72b395" containerID="d8274317608e774f10d27cc611a9c4aec293fc5e7eb16ed7e3e0fe20c3832878" exitCode=0
Nov 03 20:37:08.170578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.170309    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" event={"ID":"7205a658-0441-4700-a96e-8323bf72b395","Type":"ContainerDied","Data":"d8274317608e774f10d27cc611a9c4aec293fc5e7eb16ed7e3e0fe20c3832878"}
Nov 03 20:37:08.170578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.170337    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7" event={"ID":"7205a658-0441-4700-a96e-8323bf72b395","Type":"ContainerDied","Data":"5654181d4790c89c83d245be9a572c3305f9c0cef1182d96d31f86af4beb9fdf"}
Nov 03 20:37:08.170578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.170394    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"
Nov 03 20:37:08.178504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.178366    5659 scope.go:122] "RemoveContainer" containerID="7aafe1e6315d850f3027f8ec212e633a96e2410bbefbfd3754202e3dab11587a"
Nov 03 20:37:08.178867 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:37:08.178844    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aafe1e6315d850f3027f8ec212e633a96e2410bbefbfd3754202e3dab11587a\": container with ID starting with 7aafe1e6315d850f3027f8ec212e633a96e2410bbefbfd3754202e3dab11587a not found: ID does not exist" containerID="7aafe1e6315d850f3027f8ec212e633a96e2410bbefbfd3754202e3dab11587a"
Nov 03 20:37:08.178944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.178875    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aafe1e6315d850f3027f8ec212e633a96e2410bbefbfd3754202e3dab11587a"} err="failed to get container status \"7aafe1e6315d850f3027f8ec212e633a96e2410bbefbfd3754202e3dab11587a\": rpc error: code = NotFound desc = could not find container \"7aafe1e6315d850f3027f8ec212e633a96e2410bbefbfd3754202e3dab11587a\": container with ID starting with 7aafe1e6315d850f3027f8ec212e633a96e2410bbefbfd3754202e3dab11587a not found: ID does not exist"
Nov 03 20:37:08.178944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.178893    5659 scope.go:122] "RemoveContainer" containerID="a3607cb980e7c9ebf6ba92135b100a6fcc54ea0c0cc7b734dc9c22e758b71af5"
Nov 03 20:37:08.191119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.191098    5659 scope.go:122] "RemoveContainer" containerID="a3607cb980e7c9ebf6ba92135b100a6fcc54ea0c0cc7b734dc9c22e758b71af5"
Nov 03 20:37:08.191406 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:37:08.191386    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3607cb980e7c9ebf6ba92135b100a6fcc54ea0c0cc7b734dc9c22e758b71af5\": container with ID starting with a3607cb980e7c9ebf6ba92135b100a6fcc54ea0c0cc7b734dc9c22e758b71af5 not found: ID does not exist" containerID="a3607cb980e7c9ebf6ba92135b100a6fcc54ea0c0cc7b734dc9c22e758b71af5"
Nov 03 20:37:08.191547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.191416    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3607cb980e7c9ebf6ba92135b100a6fcc54ea0c0cc7b734dc9c22e758b71af5"} err="failed to get container status \"a3607cb980e7c9ebf6ba92135b100a6fcc54ea0c0cc7b734dc9c22e758b71af5\": rpc error: code = NotFound desc = could not find container \"a3607cb980e7c9ebf6ba92135b100a6fcc54ea0c0cc7b734dc9c22e758b71af5\": container with ID starting with a3607cb980e7c9ebf6ba92135b100a6fcc54ea0c0cc7b734dc9c22e758b71af5 not found: ID does not exist"
Nov 03 20:37:08.191547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.191442    5659 scope.go:122] "RemoveContainer" containerID="d8274317608e774f10d27cc611a9c4aec293fc5e7eb16ed7e3e0fe20c3832878"
Nov 03 20:37:08.194680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194654    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rjbg\" (UniqueName: \"kubernetes.io/projected/7205a658-0441-4700-a96e-8323bf72b395-kube-api-access-6rjbg\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194684    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194699    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194713    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194725    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194736    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194747    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194758    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194770    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194780    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c530754-24a3-4aa7-99cc-99bde27283b3-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194792    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66h52\" (UniqueName: \"kubernetes.io/projected/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-kube-api-access-66h52\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194804    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194816    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7205a658-0441-4700-a96e-8323bf72b395-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194828    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7205a658-0441-4700-a96e-8323bf72b395-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194839    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.194853 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194850    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c530754-24a3-4aa7-99cc-99bde27283b3-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.195580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194863    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5e3878-4eae-4f9e-8ae4-9483719cfbda-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.195580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.194875    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfs6c\" (UniqueName: \"kubernetes.io/projected/0c530754-24a3-4aa7-99cc-99bde27283b3-kube-api-access-lfs6c\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.200001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.199976    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"]
Nov 03 20:37:08.203766 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.203744    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/simulator-w0-d3d0df-kserve-6c969cf8f4-8bnn7"]
Nov 03 20:37:08.204344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.204323    5659 scope.go:122] "RemoveContainer" containerID="d8274317608e774f10d27cc611a9c4aec293fc5e7eb16ed7e3e0fe20c3832878"
Nov 03 20:37:08.204670 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:37:08.204650    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8274317608e774f10d27cc611a9c4aec293fc5e7eb16ed7e3e0fe20c3832878\": container with ID starting with d8274317608e774f10d27cc611a9c4aec293fc5e7eb16ed7e3e0fe20c3832878 not found: ID does not exist" containerID="d8274317608e774f10d27cc611a9c4aec293fc5e7eb16ed7e3e0fe20c3832878"
Nov 03 20:37:08.204754 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.204677    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8274317608e774f10d27cc611a9c4aec293fc5e7eb16ed7e3e0fe20c3832878"} err="failed to get container status \"d8274317608e774f10d27cc611a9c4aec293fc5e7eb16ed7e3e0fe20c3832878\": rpc error: code = NotFound desc = could not find container \"d8274317608e774f10d27cc611a9c4aec293fc5e7eb16ed7e3e0fe20c3832878\": container with ID starting with d8274317608e774f10d27cc611a9c4aec293fc5e7eb16ed7e3e0fe20c3832878 not found: ID does not exist"
Nov 03 20:37:08.219001 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.218974    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"]
Nov 03 20:37:08.225002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.224979    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/distinct-w0-d3d0df-kserve-78d9ff86d6-gwmvk"]
Nov 03 20:37:08.238181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.238121    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"]
Nov 03 20:37:08.242452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.242430    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/premium-w0-d3d0df-kserve-5655ffd8c5-ttc8m"]
Nov 03 20:37:08.242611 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.242467    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:08.958103 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:08.958064    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:09.094391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:09.094355    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:09.972830 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:09.972795    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" path="/var/lib/kubelet/pods/0c530754-24a3-4aa7-99cc-99bde27283b3/volumes"
Nov 03 20:37:09.973513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:09.973473    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" path="/var/lib/kubelet/pods/6e5e3878-4eae-4f9e-8ae4-9483719cfbda/volumes"
Nov 03 20:37:09.973994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:09.973973    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7205a658-0441-4700-a96e-8323bf72b395" path="/var/lib/kubelet/pods/7205a658-0441-4700-a96e-8323bf72b395/volumes"
Nov 03 20:37:10.095385 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:10.095351    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:10.844306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:10.844272    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:11.531024 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:11.530993    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:12.195220 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.195184    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 20:37:12.212565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.212526    5659 generic.go:356] "Generic (PLEG): container finished" podID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerID="c37184347e131df697cb225a8947a058c382cbf3e698a1c6ea07d302bdecffad" exitCode=0
Nov 03 20:37:12.212565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.212547    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"
Nov 03 20:37:12.212774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.212593    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" event={"ID":"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4","Type":"ContainerDied","Data":"c37184347e131df697cb225a8947a058c382cbf3e698a1c6ea07d302bdecffad"}
Nov 03 20:37:12.212774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.212666    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4" event={"ID":"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4","Type":"ContainerDied","Data":"fbcfea1ec973cd057bc7d3cd32141e5456273a3c7eaf430fafac4a300ceced94"}
Nov 03 20:37:12.212774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.212699    5659 scope.go:122] "RemoveContainer" containerID="c37184347e131df697cb225a8947a058c382cbf3e698a1c6ea07d302bdecffad"
Nov 03 20:37:12.214690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.214663    5659 generic.go:356] "Generic (PLEG): container finished" podID="f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" containerID="f5b8806ce4d79e7f1e325fcfded956cc4cae2a98d1012022f1cb9d0dc6b907fa" exitCode=0
Nov 03 20:37:12.214824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.214730    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc" event={"ID":"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4","Type":"ContainerDied","Data":"f5b8806ce4d79e7f1e325fcfded956cc4cae2a98d1012022f1cb9d0dc6b907fa"}
Nov 03 20:37:12.251543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.246063    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 20:37:12.253172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.253145    5659 scope.go:122] "RemoveContainer" containerID="4efbec75d49d6c0d85b68fec79beabc7210dc1a5eeefa26cabb2c2442ea08a31"
Nov 03 20:37:12.260664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.260631    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:12.310255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.290066    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-token\" (UniqueName: \"kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-token\") pod \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") "
Nov 03 20:37:12.310255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.290171    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-podinfo\") pod \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") "
Nov 03 20:37:12.310255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.290213    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-kube-api-access-lx6js\" (UniqueName: \"kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-kube-api-access-lx6js\") pod \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") "
Nov 03 20:37:12.310255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.290293    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-certs\") pod \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") "
Nov 03 20:37:12.310255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.290348    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-envoy\") pod \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") "
Nov 03 20:37:12.310255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.290492    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-data\") pod \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") "
Nov 03 20:37:12.310255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.290558    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-credential-socket\") pod \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") "
Nov 03 20:37:12.310255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.290600    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istiod-ca-cert\") pod \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") "
Nov 03 20:37:12.310255 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.290677    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-socket\") pod \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\" (UID: \"0168a66b-a1c3-4bf2-a921-8cdff64b3ed4\") "
Nov 03 20:37:12.311526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.311057    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-socket" pod "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" (UID: "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.319302 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.319243    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-certs" pod "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" (UID: "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.320955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.320912    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-credential-socket" pod "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" (UID: "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.321465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.321429    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-data" pod "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" (UID: "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.322389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.322352    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istiod-ca-cert" pod "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" (UID: "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:12.336298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.336063    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-token" pod "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" (UID: "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:12.342555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.341504    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-podinfo" pod "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" (UID: "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:12.351034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.350724    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-envoy" pod "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" (UID: "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.355536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.355501    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-kube-api-access-lx6js" pod "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" (UID: "0168a66b-a1c3-4bf2-a921-8cdff64b3ed4"). InnerVolumeSpecName "kube-api-access-lx6js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:12.356805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.356777    5659 scope.go:122] "RemoveContainer" containerID="c37184347e131df697cb225a8947a058c382cbf3e698a1c6ea07d302bdecffad"
Nov 03 20:37:12.357211 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:37:12.357181    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37184347e131df697cb225a8947a058c382cbf3e698a1c6ea07d302bdecffad\": container with ID starting with c37184347e131df697cb225a8947a058c382cbf3e698a1c6ea07d302bdecffad not found: ID does not exist" containerID="c37184347e131df697cb225a8947a058c382cbf3e698a1c6ea07d302bdecffad"
Nov 03 20:37:12.357317 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.357219    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37184347e131df697cb225a8947a058c382cbf3e698a1c6ea07d302bdecffad"} err="failed to get container status \"c37184347e131df697cb225a8947a058c382cbf3e698a1c6ea07d302bdecffad\": rpc error: code = NotFound desc = could not find container \"c37184347e131df697cb225a8947a058c382cbf3e698a1c6ea07d302bdecffad\": container with ID starting with c37184347e131df697cb225a8947a058c382cbf3e698a1c6ea07d302bdecffad not found: ID does not exist"
Nov 03 20:37:12.357317 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.357248    5659 scope.go:122] "RemoveContainer" containerID="4efbec75d49d6c0d85b68fec79beabc7210dc1a5eeefa26cabb2c2442ea08a31"
Nov 03 20:37:12.357558 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:37:12.357534    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4efbec75d49d6c0d85b68fec79beabc7210dc1a5eeefa26cabb2c2442ea08a31\": container with ID starting with 4efbec75d49d6c0d85b68fec79beabc7210dc1a5eeefa26cabb2c2442ea08a31 not found: ID does not exist" containerID="4efbec75d49d6c0d85b68fec79beabc7210dc1a5eeefa26cabb2c2442ea08a31"
Nov 03 20:37:12.357662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.357563    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efbec75d49d6c0d85b68fec79beabc7210dc1a5eeefa26cabb2c2442ea08a31"} err="failed to get container status \"4efbec75d49d6c0d85b68fec79beabc7210dc1a5eeefa26cabb2c2442ea08a31\": rpc error: code = NotFound desc = could not find container \"4efbec75d49d6c0d85b68fec79beabc7210dc1a5eeefa26cabb2c2442ea08a31\": container with ID starting with 4efbec75d49d6c0d85b68fec79beabc7210dc1a5eeefa26cabb2c2442ea08a31 not found: ID does not exist"
Nov 03 20:37:12.391884 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.391841    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-data\") pod \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") "
Nov 03 20:37:12.392080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.391928    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-podinfo\") pod \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") "
Nov 03 20:37:12.392080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.392001    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-credential-socket\") pod \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") "
Nov 03 20:37:12.392080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.392052    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-envoy\") pod \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") "
Nov 03 20:37:12.392265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.392082    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-certs\") pod \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") "
Nov 03 20:37:12.392265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.392152    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-kube-api-access-v8zlw\" (UniqueName: \"kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-kube-api-access-v8zlw\") pod \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") "
Nov 03 20:37:12.392265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.392227    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-token\" (UniqueName: \"kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-token\") pod \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") "
Nov 03 20:37:12.392425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.392316    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-socket\") pod \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") "
Nov 03 20:37:12.392515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.392461    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istiod-ca-cert\") pod \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\" (UID: \"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4\") "
Nov 03 20:37:12.393196 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.393169    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.393333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.393198    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx6js\" (UniqueName: \"kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-kube-api-access-lx6js\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.393333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.393216    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.393333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.393229    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.393333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.393242    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.393333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.393257    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.393333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.393271    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.393333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.393283    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.393333 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.393296    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.393821 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.393720    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istiod-ca-cert" pod "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" (UID: "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:12.394391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.394101    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-data" pod "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" (UID: "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.394391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.394296    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-credential-socket" pod "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" (UID: "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.394810 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.394753    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-certs" pod "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" (UID: "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.394910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.394823    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-socket" pod "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" (UID: "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.395919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.395861    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-envoy" pod "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" (UID: "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.396311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.396287    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-podinfo" pod "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" (UID: "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:12.397379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.397359    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-token" pod "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" (UID: "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:12.397468 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.397389    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-kube-api-access-v8zlw" pod "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" (UID: "f65ad726-8d5c-4ac2-b1f3-fe6589059ca4"). InnerVolumeSpecName "kube-api-access-v8zlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:12.495304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.495254    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8zlw\" (UniqueName: \"kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-kube-api-access-v8zlw\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.495304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.495299    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.495304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.495309    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.495640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.495319    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.495640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.495329    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.495640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.495341    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.495640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.495351    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.495640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.495359    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.495640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.495367    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.542261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.542218    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"]
Nov 03 20:37:12.548787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:12.548756    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-qgbf4"]
Nov 03 20:37:13.048132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:13.048095    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:13.225769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:13.225730    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"
Nov 03 20:37:13.225978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:13.225728    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc" event={"ID":"f65ad726-8d5c-4ac2-b1f3-fe6589059ca4","Type":"ContainerDied","Data":"04c7ee6bacae0dea5c3ad395bb32c3526517c5d189fd80468f21c9f45c96a4e0"}
Nov 03 20:37:13.225978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:13.225864    5659 scope.go:122] "RemoveContainer" containerID="f5b8806ce4d79e7f1e325fcfded956cc4cae2a98d1012022f1cb9d0dc6b907fa"
Nov 03 20:37:13.253631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:13.253601    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"]
Nov 03 20:37:13.259675 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:13.259648    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-5rzpc"]
Nov 03 20:37:13.706139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:13.706110    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:13.837926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:13.837888    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:13.975755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:13.975664    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" path="/var/lib/kubelet/pods/0168a66b-a1c3-4bf2-a921-8cdff64b3ed4/volumes"
Nov 03 20:37:13.976425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:13.976398    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" path="/var/lib/kubelet/pods/f65ad726-8d5c-4ac2-b1f3-fe6589059ca4/volumes"
Nov 03 20:37:14.484049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:14.484013    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:15.209519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:15.206939    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:15.284242 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:37:15.284198    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:37:15.481905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:15.481822    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:15.837422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:15.837339    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:16.399336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:16.399307    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:16.838967 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:16.838925    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:17.750508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:17.747329    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:18.528707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:18.528677    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:19.197496 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:19.197443    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:19.850606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:19.850571    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:20.574172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:20.574135    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:21.238463 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:21.238429    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:21.591767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:21.591687    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:21.741548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:21.741510    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:21.956339 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:21.956285    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:22.755002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:22.754967    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:23.636940 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:23.636905    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:24.630685 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:24.630651    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:25.377303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:25.377265    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:26.071311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:26.071272    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:26.872649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:26.872607    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:27.766268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:27.766212    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:28.520406 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:28.520366    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:29.160812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:29.160771    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:29.819291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:29.819245    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:30.077905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:30.077776    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:30.546368 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:30.546318    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:31.012684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:31.012650    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:31.189204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:31.189171    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:31.857476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:31.857441    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:32.497239 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:32.497205    5659 scope.go:122] "RemoveContainer" containerID="1466b9e49f1926675a97766a0ee53a5424b93d341d38bbb1383c74fbe34dc2be"
Nov 03 20:37:32.640883 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:32.640840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:33.353776 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:33.352170    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:34.390508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:34.388787    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:35.172283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:35.172247    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:36.139087 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:36.139059    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:36.961905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:36.961869    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:37.826775 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:37.826136    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:38.542865 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:38.542829    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:39.243885 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:39.243854    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:39.440942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:39.440840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:39.590545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:39.590053    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:39.789636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:39.789596    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:40.057472 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:40.057430    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"]
Nov 03 20:37:40.057817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:40.057780    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" podUID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" containerName="istio-proxy" containerID="cri-o://661b82ffaab9c686dac16c5fef008e78bfc13c0a098d8459b878c41e486d5af8" gracePeriod=30
Nov 03 20:37:40.077569 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:40.077463    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:40.362635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:40.362545    5659 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:37:40.363601 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:40.363244    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" podUID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:37:40.456447 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:40.456410    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:40.648950 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:40.648857    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:41.251471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:41.251431    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:41.944585 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:41.944551    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:42.642431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:42.642403    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:43.327619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:43.327583    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:44.019681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:44.019637    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:44.832375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:44.832338    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:45.228040 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:37:45.228004    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:37:45.382295 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.382227    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 20:37:45.425298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.425258    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-certs\") pod \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") "
Nov 03 20:37:45.425589 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.425556    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-certs" pod "bee48fa2-b463-4d56-b9e8-ec887c95ba29" (UID: "bee48fa2-b463-4d56-b9e8-ec887c95ba29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.425714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.425597    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-kube-api-access-ldlqg\" (UniqueName: \"kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-kube-api-access-ldlqg\") pod \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") "
Nov 03 20:37:45.425714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.425669    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-data\") pod \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") "
Nov 03 20:37:45.425714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.425700    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istiod-ca-cert\") pod \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") "
Nov 03 20:37:45.425906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.425854    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-token\" (UniqueName: \"kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-token\") pod \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") "
Nov 03 20:37:45.425969 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.425926    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-socket\") pod \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") "
Nov 03 20:37:45.426028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.425998    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-podinfo\") pod \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") "
Nov 03 20:37:45.426086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.426033    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-credential-socket\") pod \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") "
Nov 03 20:37:45.426086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.426080    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-envoy\") pod \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\" (UID: \"bee48fa2-b463-4d56-b9e8-ec887c95ba29\") "
Nov 03 20:37:45.426643 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.426625    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.427477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.426990    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-data" pod "bee48fa2-b463-4d56-b9e8-ec887c95ba29" (UID: "bee48fa2-b463-4d56-b9e8-ec887c95ba29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.427824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.427243    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istiod-ca-cert" pod "bee48fa2-b463-4d56-b9e8-ec887c95ba29" (UID: "bee48fa2-b463-4d56-b9e8-ec887c95ba29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:45.427951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.427607    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-credential-socket" pod "bee48fa2-b463-4d56-b9e8-ec887c95ba29" (UID: "bee48fa2-b463-4d56-b9e8-ec887c95ba29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.428366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.428341    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-socket" pod "bee48fa2-b463-4d56-b9e8-ec887c95ba29" (UID: "bee48fa2-b463-4d56-b9e8-ec887c95ba29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.430711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.429377    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-token" pod "bee48fa2-b463-4d56-b9e8-ec887c95ba29" (UID: "bee48fa2-b463-4d56-b9e8-ec887c95ba29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:45.430711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.430433    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-kube-api-access-ldlqg" pod "bee48fa2-b463-4d56-b9e8-ec887c95ba29" (UID: "bee48fa2-b463-4d56-b9e8-ec887c95ba29"). InnerVolumeSpecName "kube-api-access-ldlqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:45.430972 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.430935    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-podinfo" pod "bee48fa2-b463-4d56-b9e8-ec887c95ba29" (UID: "bee48fa2-b463-4d56-b9e8-ec887c95ba29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:45.432343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.432307    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-envoy" pod "bee48fa2-b463-4d56-b9e8-ec887c95ba29" (UID: "bee48fa2-b463-4d56-b9e8-ec887c95ba29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.528571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.528530    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.528571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.528558    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.528571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.528568    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.528571 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.528577    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.528869 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.528587    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.528869 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.528595    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldlqg\" (UniqueName: \"kubernetes.io/projected/bee48fa2-b463-4d56-b9e8-ec887c95ba29-kube-api-access-ldlqg\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.528869 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.528606    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.528869 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.528615    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee48fa2-b463-4d56-b9e8-ec887c95ba29-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.591521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.575128    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:45.713045 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.713006    5659 generic.go:356] "Generic (PLEG): container finished" podID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" containerID="661b82ffaab9c686dac16c5fef008e78bfc13c0a098d8459b878c41e486d5af8" exitCode=0
Nov 03 20:37:45.713235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.713055    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"
Nov 03 20:37:45.713235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.713090    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" event={"ID":"bee48fa2-b463-4d56-b9e8-ec887c95ba29","Type":"ContainerDied","Data":"661b82ffaab9c686dac16c5fef008e78bfc13c0a098d8459b878c41e486d5af8"}
Nov 03 20:37:45.713235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.713157    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh" event={"ID":"bee48fa2-b463-4d56-b9e8-ec887c95ba29","Type":"ContainerDied","Data":"478146c9833ecc78adc251d8a5a153a983b685be477d477414e06029903b4d01"}
Nov 03 20:37:45.713235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.713189    5659 scope.go:122] "RemoveContainer" containerID="661b82ffaab9c686dac16c5fef008e78bfc13c0a098d8459b878c41e486d5af8"
Nov 03 20:37:45.725702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.725685    5659 scope.go:122] "RemoveContainer" containerID="661b82ffaab9c686dac16c5fef008e78bfc13c0a098d8459b878c41e486d5af8"
Nov 03 20:37:45.725944 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:37:45.725924    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661b82ffaab9c686dac16c5fef008e78bfc13c0a098d8459b878c41e486d5af8\": container with ID starting with 661b82ffaab9c686dac16c5fef008e78bfc13c0a098d8459b878c41e486d5af8 not found: ID does not exist" containerID="661b82ffaab9c686dac16c5fef008e78bfc13c0a098d8459b878c41e486d5af8"
Nov 03 20:37:45.725996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.725952    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661b82ffaab9c686dac16c5fef008e78bfc13c0a098d8459b878c41e486d5af8"} err="failed to get container status \"661b82ffaab9c686dac16c5fef008e78bfc13c0a098d8459b878c41e486d5af8\": rpc error: code = NotFound desc = could not find container \"661b82ffaab9c686dac16c5fef008e78bfc13c0a098d8459b878c41e486d5af8\": container with ID starting with 661b82ffaab9c686dac16c5fef008e78bfc13c0a098d8459b878c41e486d5af8 not found: ID does not exist"
Nov 03 20:37:45.785828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.785785    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"]
Nov 03 20:37:45.806039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.805999    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-kjllh"]
Nov 03 20:37:45.972724 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:45.972621    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" path="/var/lib/kubelet/pods/bee48fa2-b463-4d56-b9e8-ec887c95ba29/volumes"
Nov 03 20:37:46.433407 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:46.433320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:47.147899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:47.147865    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:47.970187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:47.970148    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:48.108412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:48.108380    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:48.746144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:48.746110    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:49.522417 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:49.522364    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:50.184506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:50.181378    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:50.880773 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:50.880667    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:51.724241 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:51.724168    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:52.589506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:52.582431    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:53.203507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:53.203447    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:53.851560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:53.851524    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:54.577867 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:54.577835    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:55.328380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:55.328345    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:56.039476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:56.039428    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:56.766115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:56.766080    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:57.538998 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:57.538959    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:58.198186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:58.198148    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:58.862090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:58.862048    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:59.536077 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:37:59.536032    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:00.257773 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:00.257733    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:01.013787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:01.013725    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:01.846432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:01.846391    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:02.536366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:02.536332    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:03.255983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:03.255943    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:03.942467 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:03.942397    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:04.664930 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:04.664879    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:05.342814 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:05.342778    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:06.118645 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:06.118599    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:06.374418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:06.374331    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:06.692914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:06.692819    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:07.531043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:07.531009    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:08.172035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:08.171997    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:08.929392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:08.929355    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:09.648067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:09.648031    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:10.299937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:10.299520    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:11.168958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:11.168921    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:12.500505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:12.495599    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:13.420667 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:13.420595    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:13.555595 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:13.555564    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:14.235167 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:14.234997    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:15.038193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:15.038152    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:15.219798 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:38:15.219754    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:38:15.935469 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:15.935434    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:16.653582 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:16.647080    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:17.386513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:17.386088    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:18.154316 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:18.154282    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:18.851726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:18.851688    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:19.569654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:19.568082    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:20.329388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:20.329331    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:20.987070 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:20.987025    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:21.925603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:21.925566    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:22.639536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:22.639507    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:23.341858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:23.341824    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:24.045027 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:24.044990    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:24.198496 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:24.198377    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:24.592955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:24.592910    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:24.834062 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:24.834027    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:25.608506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:25.605127    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:25.741850 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:25.741811    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:26.288784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:26.288752    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:27.317311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:27.317256    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:28.054707 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:28.054646    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:28.878143 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:28.878102    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:29.556555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:29.556514    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:30.301801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:30.301761    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:30.552704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:30.552616    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"]
Nov 03 20:38:30.651905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:30.651870    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:30.786040 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:30.786010    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:31.173665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:31.173628    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:31.385755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:31.381585    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:32.045464 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:32.045419    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:32.850043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:32.849949    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:33.120195 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.120105    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"]
Nov 03 20:38:33.121166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.121138    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b9851d73-2717-41bd-8043-fb0557d4aa93" containerName="revoke-keys"
Nov 03 20:38:33.121353 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.121336    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9851d73-2717-41bd-8043-fb0557d4aa93" containerName="revoke-keys"
Nov 03 20:38:33.121456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.121445    5659 cpu_manager.go:414] "containerMap: removing container" podUID="4d5dac14-76e3-44c6-9926-44b0156716c7" containerName="payload-pre-processing"
Nov 03 20:38:33.135678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.135647    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5dac14-76e3-44c6-9926-44b0156716c7" containerName="payload-pre-processing"
Nov 03 20:38:33.135843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.135828    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" containerName="istio-proxy"
Nov 03 20:38:33.135947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.135932    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" containerName="istio-proxy"
Nov 03 20:38:33.136060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.136047    5659 cpu_manager.go:414] "containerMap: removing container" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy"
Nov 03 20:38:33.138285 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.138259    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy"
Nov 03 20:38:33.138536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.138518    5659 cpu_manager.go:414] "containerMap: removing container" podUID="80e7029d-ce9c-452a-8d22-2b7b12212c7c" containerName="maas-api"
Nov 03 20:38:33.138651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.138638    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e7029d-ce9c-452a-8d22-2b7b12212c7c" containerName="maas-api"
Nov 03 20:38:33.139052 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.139036    5659 cpu_manager.go:414] "containerMap: removing container" podUID="2f904191-5cd6-4550-ad9d-431ba37472cd" containerName="maas-api"
Nov 03 20:38:33.139162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.139149    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f904191-5cd6-4550-ad9d-431ba37472cd" containerName="maas-api"
Nov 03 20:38:33.139258 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.139247    5659 cpu_manager.go:414] "containerMap: removing container" podUID="de7c4c4d-2e02-4c99-b9e0-8f15391d7871" containerName="payload-pre-processing"
Nov 03 20:38:33.139350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.139337    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7c4c4d-2e02-4c99-b9e0-8f15391d7871" containerName="payload-pre-processing"
Nov 03 20:38:33.151602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.151574    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b9851d73-2717-41bd-8043-fb0557d4aa93" containerName="revoke-keys"
Nov 03 20:38:33.151784 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.151768    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9851d73-2717-41bd-8043-fb0557d4aa93" containerName="revoke-keys"
Nov 03 20:38:33.151881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.151871    5659 cpu_manager.go:414] "containerMap: removing container" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerName="istio-proxy"
Nov 03 20:38:33.151958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.151949    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerName="istio-proxy"
Nov 03 20:38:33.152033 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.152024    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main"
Nov 03 20:38:33.152093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.152084    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main"
Nov 03 20:38:33.152166 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.152157    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main"
Nov 03 20:38:33.152232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.152223    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main"
Nov 03 20:38:33.152309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.152300    5659 cpu_manager.go:414] "containerMap: removing container" podUID="1181f6f2-8a83-4b5f-bf64-77a556de321d" containerName="istio-proxy"
Nov 03 20:38:33.153519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.153499    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1181f6f2-8a83-4b5f-bf64-77a556de321d" containerName="istio-proxy"
Nov 03 20:38:33.153664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.153652    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6b9cc76e-d155-4572-b544-15613ed861e0" containerName="revoke-keys"
Nov 03 20:38:33.153748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.153736    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9cc76e-d155-4572-b544-15613ed861e0" containerName="revoke-keys"
Nov 03 20:38:33.153888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.153878    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b9851d73-2717-41bd-8043-fb0557d4aa93" containerName="revoke-keys"
Nov 03 20:38:33.153955 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.153947    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9851d73-2717-41bd-8043-fb0557d4aa93" containerName="revoke-keys"
Nov 03 20:38:33.154089 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.154079    5659 cpu_manager.go:414] "containerMap: removing container" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main"
Nov 03 20:38:33.154159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.154150    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main"
Nov 03 20:38:33.154242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.154233    5659 cpu_manager.go:414] "containerMap: removing container" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerName="istio-proxy"
Nov 03 20:38:33.154305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.154296    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerName="istio-proxy"
Nov 03 20:38:33.154370 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.154362    5659 cpu_manager.go:414] "containerMap: removing container" podUID="7349216a-564a-4fea-8cd8-4a8e8a6f3b2c" containerName="maas-api"
Nov 03 20:38:33.154439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.154430    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7349216a-564a-4fea-8cd8-4a8e8a6f3b2c" containerName="maas-api"
Nov 03 20:38:33.154528 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.154518    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b2fe4f76-2186-4315-bb74-db93367de753" containerName="istio-proxy"
Nov 03 20:38:33.154598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.154589    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2fe4f76-2186-4315-bb74-db93367de753" containerName="istio-proxy"
Nov 03 20:38:33.154675 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.154666    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8aca7ab0-c961-4029-9d38-5e5697f39f76" containerName="maas-api"
Nov 03 20:38:33.154739 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.154731    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aca7ab0-c961-4029-9d38-5e5697f39f76" containerName="maas-api"
Nov 03 20:38:33.154815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.154806    5659 cpu_manager.go:414] "containerMap: removing container" podUID="11bf2ea4-c852-4165-854c-ce2ad2018274" containerName="payload-processing"
Nov 03 20:38:33.154888 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.154879    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11bf2ea4-c852-4165-854c-ce2ad2018274" containerName="payload-processing"
Nov 03 20:38:33.154973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.154964    5659 cpu_manager.go:414] "containerMap: removing container" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main"
Nov 03 20:38:33.155050 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.155040    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main"
Nov 03 20:38:33.155129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.155120    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" containerName="istio-proxy"
Nov 03 20:38:33.155194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.155186    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" containerName="istio-proxy"
Nov 03 20:38:33.155275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.155255    5659 cpu_manager.go:414] "containerMap: removing container" podUID="99bedd28-f7f9-4445-9bf5-56f3a82b1df8" containerName="maas-api"
Nov 03 20:38:33.155375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.155359    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bedd28-f7f9-4445-9bf5-56f3a82b1df8" containerName="maas-api"
Nov 03 20:38:33.155471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.155456    5659 cpu_manager.go:414] "containerMap: removing container" podUID="15cce432-d79b-4f87-b5c6-63908192552d" containerName="istio-proxy"
Nov 03 20:38:33.159702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.159678    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15cce432-d79b-4f87-b5c6-63908192552d" containerName="istio-proxy"
Nov 03 20:38:33.159831 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.159819    5659 cpu_manager.go:414] "containerMap: removing container" podUID="87a6ec71-2a2b-4e6c-aab0-56512872cb10" containerName="istio-proxy"
Nov 03 20:38:33.159908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.159897    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a6ec71-2a2b-4e6c-aab0-56512872cb10" containerName="istio-proxy"
Nov 03 20:38:33.159986 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.159976    5659 cpu_manager.go:414] "containerMap: removing container" podUID="cc874822-913a-4d2d-922f-19d6095c34f9" containerName="istio-proxy"
Nov 03 20:38:33.160062 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.160052    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc874822-913a-4d2d-922f-19d6095c34f9" containerName="istio-proxy"
Nov 03 20:38:33.160142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.160133    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a351e8e7-b09d-44fd-ba3e-8c811db79f8a" containerName="payload-processing"
Nov 03 20:38:33.160219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.160209    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a351e8e7-b09d-44fd-ba3e-8c811db79f8a" containerName="payload-processing"
Nov 03 20:38:33.160649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.160631    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9851d73-2717-41bd-8043-fb0557d4aa93" containerName="revoke-keys"
Nov 03 20:38:33.161063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.161047    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5dac14-76e3-44c6-9926-44b0156716c7" containerName="payload-pre-processing"
Nov 03 20:38:33.161178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.161166    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee48fa2-b463-4d56-b9e8-ec887c95ba29" containerName="istio-proxy"
Nov 03 20:38:33.161265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.161255    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy"
Nov 03 20:38:33.161358 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.161348    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e7029d-ce9c-452a-8d22-2b7b12212c7c" containerName="maas-api"
Nov 03 20:38:33.161442 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.161432    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f904191-5cd6-4550-ad9d-431ba37472cd" containerName="maas-api"
Nov 03 20:38:33.161549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.161537    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7c4c4d-2e02-4c99-b9e0-8f15391d7871" containerName="payload-pre-processing"
Nov 03 20:38:33.161648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.161637    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9851d73-2717-41bd-8043-fb0557d4aa93" containerName="revoke-keys"
Nov 03 20:38:33.161731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.161721    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerName="istio-proxy"
Nov 03 20:38:33.161812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.161803    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5e3878-4eae-4f9e-8ae4-9483719cfbda" containerName="main"
Nov 03 20:38:33.161894 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.161884    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b309ad2b-6391-4e09-8bbc-ae2ca1bdef85" containerName="main"
Nov 03 20:38:33.161969 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.161960    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="1181f6f2-8a83-4b5f-bf64-77a556de321d" containerName="istio-proxy"
Nov 03 20:38:33.162048 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.162037    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9cc76e-d155-4572-b544-15613ed861e0" containerName="revoke-keys"
Nov 03 20:38:33.162134 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.162124    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9851d73-2717-41bd-8043-fb0557d4aa93" containerName="revoke-keys"
Nov 03 20:38:33.162223 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.162213    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c530754-24a3-4aa7-99cc-99bde27283b3" containerName="main"
Nov 03 20:38:33.162309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.162299    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="7349216a-564a-4fea-8cd8-4a8e8a6f3b2c" containerName="maas-api"
Nov 03 20:38:33.162389 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.162380    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2fe4f76-2186-4315-bb74-db93367de753" containerName="istio-proxy"
Nov 03 20:38:33.162494 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.162467    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aca7ab0-c961-4029-9d38-5e5697f39f76" containerName="maas-api"
Nov 03 20:38:33.162595 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.162583    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="11bf2ea4-c852-4165-854c-ce2ad2018274" containerName="payload-processing"
Nov 03 20:38:33.162681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.162670    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="7205a658-0441-4700-a96e-8323bf72b395" containerName="main"
Nov 03 20:38:33.162768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.162758    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f65ad726-8d5c-4ac2-b1f3-fe6589059ca4" containerName="istio-proxy"
Nov 03 20:38:33.171615 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.171589    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="99bedd28-f7f9-4445-9bf5-56f3a82b1df8" containerName="maas-api"
Nov 03 20:38:33.171803 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.171787    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="15cce432-d79b-4f87-b5c6-63908192552d" containerName="istio-proxy"
Nov 03 20:38:33.171905 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.171893    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a6ec71-2a2b-4e6c-aab0-56512872cb10" containerName="istio-proxy"
Nov 03 20:38:33.172007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.171995    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc874822-913a-4d2d-922f-19d6095c34f9" containerName="istio-proxy"
Nov 03 20:38:33.172095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.172084    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a351e8e7-b09d-44fd-ba3e-8c811db79f8a" containerName="payload-processing"
Nov 03 20:38:33.209527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.206062    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.211004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.210975    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"]
Nov 03 20:38:33.212181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.212095    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-43c674-openshift-default-dockercfg-slz8b\""
Nov 03 20:38:33.256509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.243615    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.256859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.256820    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.257233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.257209    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.257533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.257512    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.257800 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.257776    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.258049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.258027    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptljm\" (UniqueName: \"kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-kube-api-access-ptljm\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.258286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.258267    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.258565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.258544    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.258786 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.258768    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.361348 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.361309    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.361959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.361410    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.362082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.361927    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.362152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.362139    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.362995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.362968    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.363146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.362175    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.363146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.363091    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.363473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.363367    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.363583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.363566    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.363666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.363648    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.363737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.363719    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptljm\" (UniqueName: \"kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-kube-api-access-ptljm\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.363793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.363750    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.364525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.364464    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.365204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.365177    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.373310 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.368065    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.373310 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.373009    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.377139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.376914    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.383257 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.383232    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptljm\" (UniqueName: \"kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-kube-api-access-ptljm\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.613301 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.613261    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:33.838791 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:33.837743    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:34.015432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:34.015120    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"]
Nov 03 20:38:34.027104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:34.027064    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:38:34.027220 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:34.027141    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:38:34.027220 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:34.027189    5659 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:38:34.315397 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:34.315315    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5" event={"ID":"be08436f-abba-4a9d-9aa3-1f9aa05d883e","Type":"ContainerStarted","Data":"e8857a72f77126105a4fb861bc43dab83ecf9c5ae7a5c7d3e356e2986f92bb5d"}
Nov 03 20:38:34.315397 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:34.315368    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5" event={"ID":"be08436f-abba-4a9d-9aa3-1f9aa05d883e","Type":"ContainerStarted","Data":"0869b58d57a53d160cdc01eb058358dbf1b1cf429a873e964ed4c25442598314"}
Nov 03 20:38:34.348069 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:34.348011    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5" podStartSLOduration=1.347996863 podStartE2EDuration="1.347996863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:34.346645005 +0000 UTC m=+3666.501547435" watchObservedRunningTime="2026-11-03 20:38:34.347996863 +0000 UTC m=+3666.502899281"
Nov 03 20:38:34.614664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:34.614578    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:34.629991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:34.629945    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:35.205244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:35.205199    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"]
Nov 03 20:38:35.277121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:35.277088    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:35.327505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:35.326455    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf" podUID="d6fff012-3266-4752-b836-fe60b7e4fb2b" containerName="maas-api" containerID="cri-o://9637031f775870bfaa82d63e87db2137c387eed43656ee37b5f53d718f84c756" gracePeriod=30
Nov 03 20:38:35.553790 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:35.553534    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd" podUID="cce0df39-f9a8-4347-aa52-423d78661bfe" containerName="payload-pre-processing" containerID="cri-o://0db4f916868025d92718e603ee17744ee840eea09a4703873b4518a938cba925" gracePeriod=25
Nov 03 20:38:35.620694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:35.619363    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 20:38:35.620694 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:35.619424    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5" podUID="be08436f-abba-4a9d-9aa3-1f9aa05d883e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 20:38:35.622559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:35.622405    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:35.994596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:35.994561    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:38:36.122204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.122097    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-metrics-certs\") pod \"d6fff012-3266-4752-b836-fe60b7e4fb2b\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") "
Nov 03 20:38:36.122586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.122556    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6fff012-3266-4752-b836-fe60b7e4fb2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fff012-3266-4752-b836-fe60b7e4fb2b-openshift-service-ca\") pod \"d6fff012-3266-4752-b836-fe60b7e4fb2b\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") "
Nov 03 20:38:36.122725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.122626    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6fff012-3266-4752-b836-fe60b7e4fb2b-kube-api-access-gh6gz\" (UniqueName: \"kubernetes.io/projected/d6fff012-3266-4752-b836-fe60b7e4fb2b-kube-api-access-gh6gz\") pod \"d6fff012-3266-4752-b836-fe60b7e4fb2b\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") "
Nov 03 20:38:36.122816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.122728    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-maas-api-tls\") pod \"d6fff012-3266-4752-b836-fe60b7e4fb2b\" (UID: \"d6fff012-3266-4752-b836-fe60b7e4fb2b\") "
Nov 03 20:38:36.122993 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.122970    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6fff012-3266-4752-b836-fe60b7e4fb2b-openshift-service-ca" pod "d6fff012-3266-4752-b836-fe60b7e4fb2b" (UID: "d6fff012-3266-4752-b836-fe60b7e4fb2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:36.123326 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.123306    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fff012-3266-4752-b836-fe60b7e4fb2b-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:36.124443 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.124415    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-metrics-certs" pod "d6fff012-3266-4752-b836-fe60b7e4fb2b" (UID: "d6fff012-3266-4752-b836-fe60b7e4fb2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:36.124733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.124702    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fff012-3266-4752-b836-fe60b7e4fb2b-kube-api-access-gh6gz" pod "d6fff012-3266-4752-b836-fe60b7e4fb2b" (UID: "d6fff012-3266-4752-b836-fe60b7e4fb2b"). InnerVolumeSpecName "kube-api-access-gh6gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:36.124856 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.124813    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-maas-api-tls" pod "d6fff012-3266-4752-b836-fe60b7e4fb2b" (UID: "d6fff012-3266-4752-b836-fe60b7e4fb2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:36.224250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.224204    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:36.224250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.224242    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh6gz\" (UniqueName: \"kubernetes.io/projected/d6fff012-3266-4752-b836-fe60b7e4fb2b-kube-api-access-gh6gz\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:36.224250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.224253    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fff012-3266-4752-b836-fe60b7e4fb2b-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:36.381121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.381030    5659 generic.go:356] "Generic (PLEG): container finished" podID="d6fff012-3266-4752-b836-fe60b7e4fb2b" containerID="9637031f775870bfaa82d63e87db2137c387eed43656ee37b5f53d718f84c756" exitCode=0
Nov 03 20:38:36.381315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.381114    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"
Nov 03 20:38:36.381315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.381146    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf" event={"ID":"d6fff012-3266-4752-b836-fe60b7e4fb2b","Type":"ContainerDied","Data":"9637031f775870bfaa82d63e87db2137c387eed43656ee37b5f53d718f84c756"}
Nov 03 20:38:36.381315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.381207    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf" event={"ID":"d6fff012-3266-4752-b836-fe60b7e4fb2b","Type":"ContainerDied","Data":"4da13a8e47ba58a9bd57f1b317c9cfea36fd1aa3cd63176b980a39014a1d567d"}
Nov 03 20:38:36.381315 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.381237    5659 scope.go:122] "RemoveContainer" containerID="9637031f775870bfaa82d63e87db2137c387eed43656ee37b5f53d718f84c756"
Nov 03 20:38:36.394112 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.394093    5659 scope.go:122] "RemoveContainer" containerID="9637031f775870bfaa82d63e87db2137c387eed43656ee37b5f53d718f84c756"
Nov 03 20:38:36.394424 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:38:36.394393    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9637031f775870bfaa82d63e87db2137c387eed43656ee37b5f53d718f84c756\": container with ID starting with 9637031f775870bfaa82d63e87db2137c387eed43656ee37b5f53d718f84c756 not found: ID does not exist" containerID="9637031f775870bfaa82d63e87db2137c387eed43656ee37b5f53d718f84c756"
Nov 03 20:38:36.394518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.394433    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9637031f775870bfaa82d63e87db2137c387eed43656ee37b5f53d718f84c756"} err="failed to get container status \"9637031f775870bfaa82d63e87db2137c387eed43656ee37b5f53d718f84c756\": rpc error: code = NotFound desc = could not find container \"9637031f775870bfaa82d63e87db2137c387eed43656ee37b5f53d718f84c756\": container with ID starting with 9637031f775870bfaa82d63e87db2137c387eed43656ee37b5f53d718f84c756 not found: ID does not exist"
Nov 03 20:38:36.408004 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.407975    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"]
Nov 03 20:38:36.411363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.411344    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-eb69cb-96d6655f8-xc9zf"]
Nov 03 20:38:36.447420 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.446957    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:36.614807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.614771    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 20:38:36.614987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:36.614824    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5" podUID="be08436f-abba-4a9d-9aa3-1f9aa05d883e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 20:38:37.101267 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:37.101224    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:37.352570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:37.351269    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:37.484314 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:37.484271    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:37.614548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:37.614434    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 20:38:37.614548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:37.614537    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5" podUID="be08436f-abba-4a9d-9aa3-1f9aa05d883e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 20:38:37.863971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:37.863929    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:37.983999 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:37.983964    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6fff012-3266-4752-b836-fe60b7e4fb2b" path="/var/lib/kubelet/pods/d6fff012-3266-4752-b836-fe60b7e4fb2b/volumes"
Nov 03 20:38:38.614117 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:38.614079    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 20:38:38.614637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:38.614156    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5" podUID="be08436f-abba-4a9d-9aa3-1f9aa05d883e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 20:38:38.664279 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:38.664245    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:39.518142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:39.518095    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:39.614769 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:39.614733    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 20:38:39.615172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:39.614807    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5" podUID="be08436f-abba-4a9d-9aa3-1f9aa05d883e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 20:38:40.154026 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:40.153938    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:40.614682 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:40.614641    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 20:38:40.614882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:40.614726    5659 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5" podUID="be08436f-abba-4a9d-9aa3-1f9aa05d883e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 20:38:40.846262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:40.846228    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:41.294773 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:41.294731    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd" podUID="cce0df39-f9a8-4347-aa52-423d78661bfe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:38:41.537668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:41.537627    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:41.624177 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:41.624089    5659 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:41.624746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:41.624678    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:41.625664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:41.625642    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:38:42.300926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:42.300891    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:43.116594 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:43.116558    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:43.866032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:43.858855    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:44.543319 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:44.543274    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:45.219081 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:38:45.219039    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:38:45.290907 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:45.290776    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:45.991719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:45.991683    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:46.621369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:46.621332    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:47.382828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:47.382794    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:47.847971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:47.847932    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:48.134952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:48.133928    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"]
Nov 03 20:38:48.134952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:48.134184    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" podUID="992549e4-0f0c-4a0e-a170-a472285aa24f" containerName="istio-proxy" containerID="cri-o://0562e2f13fc22b13548d99578d92dbe854d9be4587231233e9183fab77224c35" gracePeriod=30
Nov 03 20:38:48.193184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:48.193144    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:48.449462 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:48.449378    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:48.993923 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:48.993884    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:49.426165 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:49.426069    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:50.030446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:50.030415    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:50.707941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:50.707900    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:51.294637 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:51.294594    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd" podUID="cce0df39-f9a8-4347-aa52-423d78661bfe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:38:51.374683 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:51.374645    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:52.086096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:52.086059    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:52.755347 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:52.755310    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:53.448893 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.448864    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:38:53.449667 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.449632    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:53.522129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.516652    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-token\" (UniqueName: \"kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-token\") pod \"992549e4-0f0c-4a0e-a170-a472285aa24f\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") "
Nov 03 20:38:53.522129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.516769    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-podinfo\") pod \"992549e4-0f0c-4a0e-a170-a472285aa24f\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") "
Nov 03 20:38:53.522129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.516834    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992549e4-0f0c-4a0e-a170-a472285aa24f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992549e4-0f0c-4a0e-a170-a472285aa24f-istiod-ca-cert\") pod \"992549e4-0f0c-4a0e-a170-a472285aa24f\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") "
Nov 03 20:38:53.522129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.516863    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-socket\") pod \"992549e4-0f0c-4a0e-a170-a472285aa24f\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") "
Nov 03 20:38:53.522129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.516919    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-certs\") pod \"992549e4-0f0c-4a0e-a170-a472285aa24f\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") "
Nov 03 20:38:53.522129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.517035    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-credential-socket\") pod \"992549e4-0f0c-4a0e-a170-a472285aa24f\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") "
Nov 03 20:38:53.522129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.517109    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-data\") pod \"992549e4-0f0c-4a0e-a170-a472285aa24f\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") "
Nov 03 20:38:53.522129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.517157    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-kube-api-access-xcpm5\" (UniqueName: \"kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-kube-api-access-xcpm5\") pod \"992549e4-0f0c-4a0e-a170-a472285aa24f\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") "
Nov 03 20:38:53.522129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.517216    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-envoy\") pod \"992549e4-0f0c-4a0e-a170-a472285aa24f\" (UID: \"992549e4-0f0c-4a0e-a170-a472285aa24f\") "
Nov 03 20:38:53.523453 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.523198    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-socket" pod "992549e4-0f0c-4a0e-a170-a472285aa24f" (UID: "992549e4-0f0c-4a0e-a170-a472285aa24f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.523792 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.523767    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992549e4-0f0c-4a0e-a170-a472285aa24f-istiod-ca-cert" pod "992549e4-0f0c-4a0e-a170-a472285aa24f" (UID: "992549e4-0f0c-4a0e-a170-a472285aa24f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:53.523892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.523784    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-credential-socket" pod "992549e4-0f0c-4a0e-a170-a472285aa24f" (UID: "992549e4-0f0c-4a0e-a170-a472285aa24f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.524135 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.524082    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-certs" pod "992549e4-0f0c-4a0e-a170-a472285aa24f" (UID: "992549e4-0f0c-4a0e-a170-a472285aa24f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.524240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.524116    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-data" pod "992549e4-0f0c-4a0e-a170-a472285aa24f" (UID: "992549e4-0f0c-4a0e-a170-a472285aa24f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.528130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.528087    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-envoy" pod "992549e4-0f0c-4a0e-a170-a472285aa24f" (UID: "992549e4-0f0c-4a0e-a170-a472285aa24f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.530022 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.528568    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-kube-api-access-xcpm5" pod "992549e4-0f0c-4a0e-a170-a472285aa24f" (UID: "992549e4-0f0c-4a0e-a170-a472285aa24f"). InnerVolumeSpecName "kube-api-access-xcpm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:53.532681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.532636    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-token" pod "992549e4-0f0c-4a0e-a170-a472285aa24f" (UID: "992549e4-0f0c-4a0e-a170-a472285aa24f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:53.535541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.535519    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-podinfo" pod "992549e4-0f0c-4a0e-a170-a472285aa24f" (UID: "992549e4-0f0c-4a0e-a170-a472285aa24f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:38:53.578636 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.578584    5659 generic.go:356] "Generic (PLEG): container finished" podID="992549e4-0f0c-4a0e-a170-a472285aa24f" containerID="0562e2f13fc22b13548d99578d92dbe854d9be4587231233e9183fab77224c35" exitCode=0
Nov 03 20:38:53.579023 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.578999    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" event={"ID":"992549e4-0f0c-4a0e-a170-a472285aa24f","Type":"ContainerDied","Data":"0562e2f13fc22b13548d99578d92dbe854d9be4587231233e9183fab77224c35"}
Nov 03 20:38:53.579168 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.579153    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b" event={"ID":"992549e4-0f0c-4a0e-a170-a472285aa24f","Type":"ContainerDied","Data":"050a9a8dfde5045e859a7afb1e8996c7dbd81f9eb0317c910711a22639401771"}
Nov 03 20:38:53.579303 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.579289    5659 scope.go:122] "RemoveContainer" containerID="0562e2f13fc22b13548d99578d92dbe854d9be4587231233e9183fab77224c35"
Nov 03 20:38:53.579597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.579579    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"
Nov 03 20:38:53.598161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.598131    5659 scope.go:122] "RemoveContainer" containerID="0562e2f13fc22b13548d99578d92dbe854d9be4587231233e9183fab77224c35"
Nov 03 20:38:53.598551 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:38:53.598524    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0562e2f13fc22b13548d99578d92dbe854d9be4587231233e9183fab77224c35\": container with ID starting with 0562e2f13fc22b13548d99578d92dbe854d9be4587231233e9183fab77224c35 not found: ID does not exist" containerID="0562e2f13fc22b13548d99578d92dbe854d9be4587231233e9183fab77224c35"
Nov 03 20:38:53.598655 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.598560    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0562e2f13fc22b13548d99578d92dbe854d9be4587231233e9183fab77224c35"} err="failed to get container status \"0562e2f13fc22b13548d99578d92dbe854d9be4587231233e9183fab77224c35\": rpc error: code = NotFound desc = could not find container \"0562e2f13fc22b13548d99578d92dbe854d9be4587231233e9183fab77224c35\": container with ID starting with 0562e2f13fc22b13548d99578d92dbe854d9be4587231233e9183fab77224c35 not found: ID does not exist"
Nov 03 20:38:53.610718 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.610677    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"]
Nov 03 20:38:53.615130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.615096    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-s9d4b"]
Nov 03 20:38:53.620067 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.620034    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.620186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.620071    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.620186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.620086    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.620186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.620101    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcpm5\" (UniqueName: \"kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-kube-api-access-xcpm5\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.620186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.620117    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.620186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.620132    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.620186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.620145    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/992549e4-0f0c-4a0e-a170-a472285aa24f-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.620186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.620160    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/992549e4-0f0c-4a0e-a170-a472285aa24f-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.620186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.620175    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/992549e4-0f0c-4a0e-a170-a472285aa24f-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.970977 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:53.970941    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992549e4-0f0c-4a0e-a170-a472285aa24f" path="/var/lib/kubelet/pods/992549e4-0f0c-4a0e-a170-a472285aa24f/volumes"
Nov 03 20:38:54.255366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:54.255272    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:55.133002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:55.132959    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:55.197233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:55.196297    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"]
Nov 03 20:38:55.208228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:55.208192    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"]
Nov 03 20:38:55.303716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:55.303678    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:55.431298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:55.431204    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:55.876586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:55.876550    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"
Nov 03 20:38:55.955008 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:55.954964    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:56.015554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:55.995955    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce0df39-f9a8-4347-aa52-423d78661bfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce0df39-f9a8-4347-aa52-423d78661bfe-plugins-config-volume\") pod \"cce0df39-f9a8-4347-aa52-423d78661bfe\" (UID: \"cce0df39-f9a8-4347-aa52-423d78661bfe\") "
Nov 03 20:38:56.015554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:55.996014    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce0df39-f9a8-4347-aa52-423d78661bfe-kube-api-access-976wz\" (UniqueName: \"kubernetes.io/projected/cce0df39-f9a8-4347-aa52-423d78661bfe-kube-api-access-976wz\") pod \"cce0df39-f9a8-4347-aa52-423d78661bfe\" (UID: \"cce0df39-f9a8-4347-aa52-423d78661bfe\") "
Nov 03 20:38:56.015554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.012543    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce0df39-f9a8-4347-aa52-423d78661bfe-plugins-config-volume" pod "cce0df39-f9a8-4347-aa52-423d78661bfe" (UID: "cce0df39-f9a8-4347-aa52-423d78661bfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:56.038686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.038647    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce0df39-f9a8-4347-aa52-423d78661bfe-kube-api-access-976wz" pod "cce0df39-f9a8-4347-aa52-423d78661bfe" (UID: "cce0df39-f9a8-4347-aa52-423d78661bfe"). InnerVolumeSpecName "kube-api-access-976wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:56.112866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.112793    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cce0df39-f9a8-4347-aa52-423d78661bfe-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:56.112866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.112825    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-976wz\" (UniqueName: \"kubernetes.io/projected/cce0df39-f9a8-4347-aa52-423d78661bfe-kube-api-access-976wz\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:56.635317 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.635272    5659 generic.go:356] "Generic (PLEG): container finished" podID="cce0df39-f9a8-4347-aa52-423d78661bfe" containerID="0db4f916868025d92718e603ee17744ee840eea09a4703873b4518a938cba925" exitCode=0
Nov 03 20:38:56.635891 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.635362    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd" event={"ID":"cce0df39-f9a8-4347-aa52-423d78661bfe","Type":"ContainerDied","Data":"0db4f916868025d92718e603ee17744ee840eea09a4703873b4518a938cba925"}
Nov 03 20:38:56.635891 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.635432    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd" event={"ID":"cce0df39-f9a8-4347-aa52-423d78661bfe","Type":"ContainerDied","Data":"8238d9a753f4950c76b97efba740e3811ff872d02882ebfb0f680a0d5879efed"}
Nov 03 20:38:56.635891 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.635465    5659 scope.go:122] "RemoveContainer" containerID="0db4f916868025d92718e603ee17744ee840eea09a4703873b4518a938cba925"
Nov 03 20:38:56.636184 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.636156    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"
Nov 03 20:38:56.649829 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.649579    5659 scope.go:122] "RemoveContainer" containerID="0db4f916868025d92718e603ee17744ee840eea09a4703873b4518a938cba925"
Nov 03 20:38:56.649952 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:38:56.649930    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db4f916868025d92718e603ee17744ee840eea09a4703873b4518a938cba925\": container with ID starting with 0db4f916868025d92718e603ee17744ee840eea09a4703873b4518a938cba925 not found: ID does not exist" containerID="0db4f916868025d92718e603ee17744ee840eea09a4703873b4518a938cba925"
Nov 03 20:38:56.650023 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.649959    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db4f916868025d92718e603ee17744ee840eea09a4703873b4518a938cba925"} err="failed to get container status \"0db4f916868025d92718e603ee17744ee840eea09a4703873b4518a938cba925\": rpc error: code = NotFound desc = could not find container \"0db4f916868025d92718e603ee17744ee840eea09a4703873b4518a938cba925\": container with ID starting with 0db4f916868025d92718e603ee17744ee840eea09a4703873b4518a938cba925 not found: ID does not exist"
Nov 03 20:38:56.667777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.667742    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"]
Nov 03 20:38:56.681778 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.681734    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-eb69cb-59567676d7-97lcd"]
Nov 03 20:38:56.774501 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:56.773459    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:57.017513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:57.015056    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:57.611633 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:57.611596    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:57.614350 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:57.613955    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"]
Nov 03 20:38:57.683827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:57.683773    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879" podUID="821779e9-42f1-4acc-8771-39d4e55f9abf" containerName="maas-api" containerID="cri-o://dbea45d6082933388ab4950cf968cfdc64349c709b4c710b4f6ef10bde67e4b0" gracePeriod=30
Nov 03 20:38:57.973595 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:57.970427    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:58.080996 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.079402    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce0df39-f9a8-4347-aa52-423d78661bfe" path="/var/lib/kubelet/pods/cce0df39-f9a8-4347-aa52-423d78661bfe/volumes"
Nov 03 20:38:58.099944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.099879    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:58.343396 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.343362    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:38:58.487147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.486810    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-maas-api-tls\") pod \"821779e9-42f1-4acc-8771-39d4e55f9abf\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") "
Nov 03 20:38:58.488469 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.488444    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-metrics-certs\") pod \"821779e9-42f1-4acc-8771-39d4e55f9abf\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") "
Nov 03 20:38:58.488791 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.488772    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821779e9-42f1-4acc-8771-39d4e55f9abf-kube-api-access-g2hpd\" (UniqueName: \"kubernetes.io/projected/821779e9-42f1-4acc-8771-39d4e55f9abf-kube-api-access-g2hpd\") pod \"821779e9-42f1-4acc-8771-39d4e55f9abf\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") "
Nov 03 20:38:58.489318 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.489298    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821779e9-42f1-4acc-8771-39d4e55f9abf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821779e9-42f1-4acc-8771-39d4e55f9abf-openshift-service-ca\") pod \"821779e9-42f1-4acc-8771-39d4e55f9abf\" (UID: \"821779e9-42f1-4acc-8771-39d4e55f9abf\") "
Nov 03 20:38:58.490128 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.489856    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821779e9-42f1-4acc-8771-39d4e55f9abf-openshift-service-ca" pod "821779e9-42f1-4acc-8771-39d4e55f9abf" (UID: "821779e9-42f1-4acc-8771-39d4e55f9abf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:58.490285 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.490265    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-maas-api-tls" pod "821779e9-42f1-4acc-8771-39d4e55f9abf" (UID: "821779e9-42f1-4acc-8771-39d4e55f9abf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:58.490892 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.490865    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821779e9-42f1-4acc-8771-39d4e55f9abf-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:58.491078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.491061    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:58.491338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.491304    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-metrics-certs" pod "821779e9-42f1-4acc-8771-39d4e55f9abf" (UID: "821779e9-42f1-4acc-8771-39d4e55f9abf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:58.492634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.492598    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821779e9-42f1-4acc-8771-39d4e55f9abf-kube-api-access-g2hpd" pod "821779e9-42f1-4acc-8771-39d4e55f9abf" (UID: "821779e9-42f1-4acc-8771-39d4e55f9abf"). InnerVolumeSpecName "kube-api-access-g2hpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:58.592610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.592569    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/821779e9-42f1-4acc-8771-39d4e55f9abf-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:58.592610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.592611    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2hpd\" (UniqueName: \"kubernetes.io/projected/821779e9-42f1-4acc-8771-39d4e55f9abf-kube-api-access-g2hpd\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:58.640228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.640187    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:58.718982 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.718944    5659 generic.go:356] "Generic (PLEG): container finished" podID="821779e9-42f1-4acc-8771-39d4e55f9abf" containerID="dbea45d6082933388ab4950cf968cfdc64349c709b4c710b4f6ef10bde67e4b0" exitCode=0
Nov 03 20:38:58.719497 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.718996    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"
Nov 03 20:38:58.719497 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.719041    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879" event={"ID":"821779e9-42f1-4acc-8771-39d4e55f9abf","Type":"ContainerDied","Data":"dbea45d6082933388ab4950cf968cfdc64349c709b4c710b4f6ef10bde67e4b0"}
Nov 03 20:38:58.719497 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.719112    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879" event={"ID":"821779e9-42f1-4acc-8771-39d4e55f9abf","Type":"ContainerDied","Data":"028621ac3ce088b039131168d66624b61e7928bf0d7de6efcf5fa0df098525ea"}
Nov 03 20:38:58.719497 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.719146    5659 scope.go:122] "RemoveContainer" containerID="dbea45d6082933388ab4950cf968cfdc64349c709b4c710b4f6ef10bde67e4b0"
Nov 03 20:38:58.732708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.732689    5659 scope.go:122] "RemoveContainer" containerID="dbea45d6082933388ab4950cf968cfdc64349c709b4c710b4f6ef10bde67e4b0"
Nov 03 20:38:58.733039 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:38:58.733018    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbea45d6082933388ab4950cf968cfdc64349c709b4c710b4f6ef10bde67e4b0\": container with ID starting with dbea45d6082933388ab4950cf968cfdc64349c709b4c710b4f6ef10bde67e4b0 not found: ID does not exist" containerID="dbea45d6082933388ab4950cf968cfdc64349c709b4c710b4f6ef10bde67e4b0"
Nov 03 20:38:58.733126 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.733046    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbea45d6082933388ab4950cf968cfdc64349c709b4c710b4f6ef10bde67e4b0"} err="failed to get container status \"dbea45d6082933388ab4950cf968cfdc64349c709b4c710b4f6ef10bde67e4b0\": rpc error: code = NotFound desc = could not find container \"dbea45d6082933388ab4950cf968cfdc64349c709b4c710b4f6ef10bde67e4b0\": container with ID starting with dbea45d6082933388ab4950cf968cfdc64349c709b4c710b4f6ef10bde67e4b0 not found: ID does not exist"
Nov 03 20:38:58.744409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.744349    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"]
Nov 03 20:38:58.750134 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:58.750111    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-46a6d7-747f987c74-6j879"]
Nov 03 20:38:59.040725 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:59.040627    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:59.460070 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:59.460036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:59.582763 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:59.582729    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:59.973517 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:38:59.973467    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821779e9-42f1-4acc-8771-39d4e55f9abf" path="/var/lib/kubelet/pods/821779e9-42f1-4acc-8771-39d4e55f9abf/volumes"
Nov 03 20:39:00.197039 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.196995    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg" podUID="c981a786-15c1-4282-9823-8a937781f2fe" containerName="payload-processing" containerID="cri-o://d94f75e6653a403e2071874c0a6b1d2c080be663db7b0b4eeb44620ef7dd9a16" gracePeriod=25
Nov 03 20:39:00.208827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.208789    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229" podUID="7c195f09-b9ef-45a0-89d6-07ea2a203277" containerName="payload-pre-processing" containerID="cri-o://f90c901c67e88f92bff67effb2025c35dd61e7f28af42d1cf65da1825123dc68" gracePeriod=25
Nov 03 20:39:00.483608 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.480909    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:00.587547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.587151    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"
Nov 03 20:39:00.614664 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.614099    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"
Nov 03 20:39:00.649947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.649916    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c981a786-15c1-4282-9823-8a937781f2fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c981a786-15c1-4282-9823-8a937781f2fe-plugins-config-volume\") pod \"c981a786-15c1-4282-9823-8a937781f2fe\" (UID: \"c981a786-15c1-4282-9823-8a937781f2fe\") "
Nov 03 20:39:00.650963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.650934    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c981a786-15c1-4282-9823-8a937781f2fe-kube-api-access-9z78r\" (UniqueName: \"kubernetes.io/projected/c981a786-15c1-4282-9823-8a937781f2fe-kube-api-access-9z78r\") pod \"c981a786-15c1-4282-9823-8a937781f2fe\" (UID: \"c981a786-15c1-4282-9823-8a937781f2fe\") "
Nov 03 20:39:00.651182 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.651155    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c195f09-b9ef-45a0-89d6-07ea2a203277-kube-api-access-sbpkj\" (UniqueName: \"kubernetes.io/projected/7c195f09-b9ef-45a0-89d6-07ea2a203277-kube-api-access-sbpkj\") pod \"7c195f09-b9ef-45a0-89d6-07ea2a203277\" (UID: \"7c195f09-b9ef-45a0-89d6-07ea2a203277\") "
Nov 03 20:39:00.652016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.650827    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c981a786-15c1-4282-9823-8a937781f2fe-plugins-config-volume" pod "c981a786-15c1-4282-9823-8a937781f2fe" (UID: "c981a786-15c1-4282-9823-8a937781f2fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:00.652788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.652737    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c195f09-b9ef-45a0-89d6-07ea2a203277-plugins-config-volume" pod "7c195f09-b9ef-45a0-89d6-07ea2a203277" (UID: "7c195f09-b9ef-45a0-89d6-07ea2a203277"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:00.652943 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.652271    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c195f09-b9ef-45a0-89d6-07ea2a203277-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c195f09-b9ef-45a0-89d6-07ea2a203277-plugins-config-volume\") pod \"7c195f09-b9ef-45a0-89d6-07ea2a203277\" (UID: \"7c195f09-b9ef-45a0-89d6-07ea2a203277\") "
Nov 03 20:39:00.658288 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.658268    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c195f09-b9ef-45a0-89d6-07ea2a203277-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:00.658408 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.658397    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c981a786-15c1-4282-9823-8a937781f2fe-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:00.658812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.658762    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c195f09-b9ef-45a0-89d6-07ea2a203277-kube-api-access-sbpkj" pod "7c195f09-b9ef-45a0-89d6-07ea2a203277" (UID: "7c195f09-b9ef-45a0-89d6-07ea2a203277"). InnerVolumeSpecName "kube-api-access-sbpkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:00.660543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.660467    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c981a786-15c1-4282-9823-8a937781f2fe-kube-api-access-9z78r" pod "c981a786-15c1-4282-9823-8a937781f2fe" (UID: "c981a786-15c1-4282-9823-8a937781f2fe"). InnerVolumeSpecName "kube-api-access-9z78r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:00.737274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.737185    5659 generic.go:356] "Generic (PLEG): container finished" podID="c981a786-15c1-4282-9823-8a937781f2fe" containerID="d94f75e6653a403e2071874c0a6b1d2c080be663db7b0b4eeb44620ef7dd9a16" exitCode=0
Nov 03 20:39:00.737274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.737253    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"
Nov 03 20:39:00.737274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.737257    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg" event={"ID":"c981a786-15c1-4282-9823-8a937781f2fe","Type":"ContainerDied","Data":"d94f75e6653a403e2071874c0a6b1d2c080be663db7b0b4eeb44620ef7dd9a16"}
Nov 03 20:39:00.737578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.737349    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg" event={"ID":"c981a786-15c1-4282-9823-8a937781f2fe","Type":"ContainerDied","Data":"31de108f82abca9f8b4035c65f02953538518eb36c4a0e740e3b0b059c792e3f"}
Nov 03 20:39:00.737578 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.737379    5659 scope.go:122] "RemoveContainer" containerID="d94f75e6653a403e2071874c0a6b1d2c080be663db7b0b4eeb44620ef7dd9a16"
Nov 03 20:39:00.739268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.739236    5659 generic.go:356] "Generic (PLEG): container finished" podID="7c195f09-b9ef-45a0-89d6-07ea2a203277" containerID="f90c901c67e88f92bff67effb2025c35dd61e7f28af42d1cf65da1825123dc68" exitCode=0
Nov 03 20:39:00.739391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.739331    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229" event={"ID":"7c195f09-b9ef-45a0-89d6-07ea2a203277","Type":"ContainerDied","Data":"f90c901c67e88f92bff67effb2025c35dd61e7f28af42d1cf65da1825123dc68"}
Nov 03 20:39:00.739391 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.739377    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229" event={"ID":"7c195f09-b9ef-45a0-89d6-07ea2a203277","Type":"ContainerDied","Data":"3ccfd15dcb235eaa430b51b267de518f508c201343c794478a4f63019dc478f4"}
Nov 03 20:39:00.739545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.739446    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"
Nov 03 20:39:00.751419 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.751391    5659 scope.go:122] "RemoveContainer" containerID="d94f75e6653a403e2071874c0a6b1d2c080be663db7b0b4eeb44620ef7dd9a16"
Nov 03 20:39:00.751905 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:00.751883    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94f75e6653a403e2071874c0a6b1d2c080be663db7b0b4eeb44620ef7dd9a16\": container with ID starting with d94f75e6653a403e2071874c0a6b1d2c080be663db7b0b4eeb44620ef7dd9a16 not found: ID does not exist" containerID="d94f75e6653a403e2071874c0a6b1d2c080be663db7b0b4eeb44620ef7dd9a16"
Nov 03 20:39:00.752032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.751917    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94f75e6653a403e2071874c0a6b1d2c080be663db7b0b4eeb44620ef7dd9a16"} err="failed to get container status \"d94f75e6653a403e2071874c0a6b1d2c080be663db7b0b4eeb44620ef7dd9a16\": rpc error: code = NotFound desc = could not find container \"d94f75e6653a403e2071874c0a6b1d2c080be663db7b0b4eeb44620ef7dd9a16\": container with ID starting with d94f75e6653a403e2071874c0a6b1d2c080be663db7b0b4eeb44620ef7dd9a16 not found: ID does not exist"
Nov 03 20:39:00.752032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.751940    5659 scope.go:122] "RemoveContainer" containerID="f90c901c67e88f92bff67effb2025c35dd61e7f28af42d1cf65da1825123dc68"
Nov 03 20:39:00.760238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.760211    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbpkj\" (UniqueName: \"kubernetes.io/projected/7c195f09-b9ef-45a0-89d6-07ea2a203277-kube-api-access-sbpkj\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:00.760347 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.760243    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z78r\" (UniqueName: \"kubernetes.io/projected/c981a786-15c1-4282-9823-8a937781f2fe-kube-api-access-9z78r\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:00.768494 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.768461    5659 scope.go:122] "RemoveContainer" containerID="f90c901c67e88f92bff67effb2025c35dd61e7f28af42d1cf65da1825123dc68"
Nov 03 20:39:00.768837 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:00.768817    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90c901c67e88f92bff67effb2025c35dd61e7f28af42d1cf65da1825123dc68\": container with ID starting with f90c901c67e88f92bff67effb2025c35dd61e7f28af42d1cf65da1825123dc68 not found: ID does not exist" containerID="f90c901c67e88f92bff67effb2025c35dd61e7f28af42d1cf65da1825123dc68"
Nov 03 20:39:00.768915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.768845    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90c901c67e88f92bff67effb2025c35dd61e7f28af42d1cf65da1825123dc68"} err="failed to get container status \"f90c901c67e88f92bff67effb2025c35dd61e7f28af42d1cf65da1825123dc68\": rpc error: code = NotFound desc = could not find container \"f90c901c67e88f92bff67effb2025c35dd61e7f28af42d1cf65da1825123dc68\": container with ID starting with f90c901c67e88f92bff67effb2025c35dd61e7f28af42d1cf65da1825123dc68 not found: ID does not exist"
Nov 03 20:39:00.777638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.777581    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"]
Nov 03 20:39:00.781521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.781471    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-46a6d7-fbf745886-grqvg"]
Nov 03 20:39:00.793161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.793077    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"]
Nov 03 20:39:00.797572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:00.797543    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-46a6d7-86dcdbbcf9-sl229"]
Nov 03 20:39:01.217270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:01.217234    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:01.972035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:01.971999    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c195f09-b9ef-45a0-89d6-07ea2a203277" path="/var/lib/kubelet/pods/7c195f09-b9ef-45a0-89d6-07ea2a203277/volumes"
Nov 03 20:39:01.972476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:01.972461    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c981a786-15c1-4282-9823-8a937781f2fe" path="/var/lib/kubelet/pods/c981a786-15c1-4282-9823-8a937781f2fe/volumes"
Nov 03 20:39:02.054958 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:02.054920    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:02.966380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:02.966310    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:03.835821 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:03.835783    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:04.006651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:04.006612    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:04.145474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:04.145352    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:04.491882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:04.491852    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:04.648019 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:04.647984    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:04.823909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:04.823824    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"]
Nov 03 20:39:04.824421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:04.824389    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" podUID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" containerName="istio-proxy" containerID="cri-o://644f3389cf12ca2ff17fb4d8d4e18752e106e72b6619d1aa5fed451577ea0494" gracePeriod=30
Nov 03 20:39:04.944106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:04.944070    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:05.081021 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:05.080920    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:05.638249 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:05.638219    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:05.798626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:05.798585    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:05.861644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:05.861587    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"]
Nov 03 20:39:05.873525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:05.873462    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"]
Nov 03 20:39:05.951385 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:05.951247    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:06.198059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:06.197977    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:06.438503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:06.436688    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:06.777612 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:06.767518    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:07.351529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:07.351469    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:07.538975 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:07.538934    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:07.935444 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:07.935410    5659 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:39:07.935646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:07.935470    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" podUID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:39:08.335104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:08.335067    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:09.092519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:09.092427    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:09.677099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:09.677062    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:09.853329 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:09.853298    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:10.131139 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.131110    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:10.350227 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.350198    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:39:10.437235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.437140    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-certs\") pod \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") "
Nov 03 20:39:10.437235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.437202    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-podinfo\") pod \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") "
Nov 03 20:39:10.437235 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.437232    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-socket\") pod \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") "
Nov 03 20:39:10.437580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.437258    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istiod-ca-cert\") pod \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") "
Nov 03 20:39:10.437580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.437394    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-token\" (UniqueName: \"kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-token\") pod \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") "
Nov 03 20:39:10.437580 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.437520    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-certs" pod "72c08e50-44dd-4bf9-ae14-f24e2672e4c5" (UID: "72c08e50-44dd-4bf9-ae14-f24e2672e4c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:10.437764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.437592    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-socket" pod "72c08e50-44dd-4bf9-ae14-f24e2672e4c5" (UID: "72c08e50-44dd-4bf9-ae14-f24e2672e4c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:10.437764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.437636    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-data\") pod \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") "
Nov 03 20:39:10.437764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.437707    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-kube-api-access-cb59k\" (UniqueName: \"kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-kube-api-access-cb59k\") pod \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") "
Nov 03 20:39:10.437764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.437757    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-credential-socket\") pod \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") "
Nov 03 20:39:10.437987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.437831    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-envoy\") pod \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\" (UID: \"72c08e50-44dd-4bf9-ae14-f24e2672e4c5\") "
Nov 03 20:39:10.438109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.437855    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-data" pod "72c08e50-44dd-4bf9-ae14-f24e2672e4c5" (UID: "72c08e50-44dd-4bf9-ae14-f24e2672e4c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:10.438207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.438185    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istiod-ca-cert" pod "72c08e50-44dd-4bf9-ae14-f24e2672e4c5" (UID: "72c08e50-44dd-4bf9-ae14-f24e2672e4c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:10.438330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.438302    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-credential-socket" pod "72c08e50-44dd-4bf9-ae14-f24e2672e4c5" (UID: "72c08e50-44dd-4bf9-ae14-f24e2672e4c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:10.438732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.438704    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.438732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.438731    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.438901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.438745    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.438901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.438755    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.438901 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.438765    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.440286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.440249    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-envoy" pod "72c08e50-44dd-4bf9-ae14-f24e2672e4c5" (UID: "72c08e50-44dd-4bf9-ae14-f24e2672e4c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:10.440397 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.440301    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-podinfo" pod "72c08e50-44dd-4bf9-ae14-f24e2672e4c5" (UID: "72c08e50-44dd-4bf9-ae14-f24e2672e4c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:39:10.440397 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.440354    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-token" pod "72c08e50-44dd-4bf9-ae14-f24e2672e4c5" (UID: "72c08e50-44dd-4bf9-ae14-f24e2672e4c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:10.440815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.440795    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-kube-api-access-cb59k" pod "72c08e50-44dd-4bf9-ae14-f24e2672e4c5" (UID: "72c08e50-44dd-4bf9-ae14-f24e2672e4c5"). InnerVolumeSpecName "kube-api-access-cb59k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:10.540470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.540430    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.540470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.540463    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.540470 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.540474    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.540738 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.540501    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb59k\" (UniqueName: \"kubernetes.io/projected/72c08e50-44dd-4bf9-ae14-f24e2672e4c5-kube-api-access-cb59k\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.651096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.650965    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:10.863191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.863111    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd" podUID="a7356100-2a57-4289-b38f-133d56fa44d1" containerName="payload-processing" containerID="cri-o://a61626e49efa7334c7e4ab289ee564a4b49614c796efd0a4f04bd466ef945502" gracePeriod=25
Nov 03 20:39:10.874793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.874748    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58" podUID="bfcbe9c0-443c-4551-93f9-1b50010ff9f0" containerName="payload-pre-processing" containerID="cri-o://ae9e88d90a6c9997803e33594a2c4fdcc640016ef80923217df8ff6fcc6fa0db" gracePeriod=25
Nov 03 20:39:10.920959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.920921    5659 generic.go:356] "Generic (PLEG): container finished" podID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" containerID="644f3389cf12ca2ff17fb4d8d4e18752e106e72b6619d1aa5fed451577ea0494" exitCode=0
Nov 03 20:39:10.921104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.920973    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"
Nov 03 20:39:10.921104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.920999    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" event={"ID":"72c08e50-44dd-4bf9-ae14-f24e2672e4c5","Type":"ContainerDied","Data":"644f3389cf12ca2ff17fb4d8d4e18752e106e72b6619d1aa5fed451577ea0494"}
Nov 03 20:39:10.921104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.921059    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q" event={"ID":"72c08e50-44dd-4bf9-ae14-f24e2672e4c5","Type":"ContainerDied","Data":"6f806608af35668d035c9e7519f13094bfcb3d5ac48c59b95527c5cebc162652"}
Nov 03 20:39:10.921104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.921084    5659 scope.go:122] "RemoveContainer" containerID="644f3389cf12ca2ff17fb4d8d4e18752e106e72b6619d1aa5fed451577ea0494"
Nov 03 20:39:10.982828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.982788    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"]
Nov 03 20:39:10.989381 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.989349    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-xbz4q"]
Nov 03 20:39:10.992423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.992399    5659 scope.go:122] "RemoveContainer" containerID="644f3389cf12ca2ff17fb4d8d4e18752e106e72b6619d1aa5fed451577ea0494"
Nov 03 20:39:10.992800 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:10.992781    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644f3389cf12ca2ff17fb4d8d4e18752e106e72b6619d1aa5fed451577ea0494\": container with ID starting with 644f3389cf12ca2ff17fb4d8d4e18752e106e72b6619d1aa5fed451577ea0494 not found: ID does not exist" containerID="644f3389cf12ca2ff17fb4d8d4e18752e106e72b6619d1aa5fed451577ea0494"
Nov 03 20:39:10.992900 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:10.992814    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644f3389cf12ca2ff17fb4d8d4e18752e106e72b6619d1aa5fed451577ea0494"} err="failed to get container status \"644f3389cf12ca2ff17fb4d8d4e18752e106e72b6619d1aa5fed451577ea0494\": rpc error: code = NotFound desc = could not find container \"644f3389cf12ca2ff17fb4d8d4e18752e106e72b6619d1aa5fed451577ea0494\": container with ID starting with 644f3389cf12ca2ff17fb4d8d4e18752e106e72b6619d1aa5fed451577ea0494 not found: ID does not exist"
Nov 03 20:39:11.197987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.197951    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"
Nov 03 20:39:11.236910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.236876    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"
Nov 03 20:39:11.368761 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.368661    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:11.386631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.383872    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-plugins-config-volume\") pod \"bfcbe9c0-443c-4551-93f9-1b50010ff9f0\" (UID: \"bfcbe9c0-443c-4551-93f9-1b50010ff9f0\") "
Nov 03 20:39:11.386631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.383950    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-kube-api-access-xw8xd\" (UniqueName: \"kubernetes.io/projected/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-kube-api-access-xw8xd\") pod \"bfcbe9c0-443c-4551-93f9-1b50010ff9f0\" (UID: \"bfcbe9c0-443c-4551-93f9-1b50010ff9f0\") "
Nov 03 20:39:11.386631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.384004    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7356100-2a57-4289-b38f-133d56fa44d1-kube-api-access-95zbk\" (UniqueName: \"kubernetes.io/projected/a7356100-2a57-4289-b38f-133d56fa44d1-kube-api-access-95zbk\") pod \"a7356100-2a57-4289-b38f-133d56fa44d1\" (UID: \"a7356100-2a57-4289-b38f-133d56fa44d1\") "
Nov 03 20:39:11.386631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.384046    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7356100-2a57-4289-b38f-133d56fa44d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7356100-2a57-4289-b38f-133d56fa44d1-plugins-config-volume\") pod \"a7356100-2a57-4289-b38f-133d56fa44d1\" (UID: \"a7356100-2a57-4289-b38f-133d56fa44d1\") "
Nov 03 20:39:11.386631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.385491    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7356100-2a57-4289-b38f-133d56fa44d1-plugins-config-volume" pod "a7356100-2a57-4289-b38f-133d56fa44d1" (UID: "a7356100-2a57-4289-b38f-133d56fa44d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:11.386631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.385541    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-plugins-config-volume" pod "bfcbe9c0-443c-4551-93f9-1b50010ff9f0" (UID: "bfcbe9c0-443c-4551-93f9-1b50010ff9f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:11.399807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.399141    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-kube-api-access-xw8xd" pod "bfcbe9c0-443c-4551-93f9-1b50010ff9f0" (UID: "bfcbe9c0-443c-4551-93f9-1b50010ff9f0"). InnerVolumeSpecName "kube-api-access-xw8xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:11.399807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.399242    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7356100-2a57-4289-b38f-133d56fa44d1-kube-api-access-95zbk" pod "a7356100-2a57-4289-b38f-133d56fa44d1" (UID: "a7356100-2a57-4289-b38f-133d56fa44d1"). InnerVolumeSpecName "kube-api-access-95zbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:11.486292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.486252    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:11.486292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.486284    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw8xd\" (UniqueName: \"kubernetes.io/projected/bfcbe9c0-443c-4551-93f9-1b50010ff9f0-kube-api-access-xw8xd\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:11.486292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.486294    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95zbk\" (UniqueName: \"kubernetes.io/projected/a7356100-2a57-4289-b38f-133d56fa44d1-kube-api-access-95zbk\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:11.486541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.486304    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7356100-2a57-4289-b38f-133d56fa44d1-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:11.932843 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.932805    5659 generic.go:356] "Generic (PLEG): container finished" podID="bfcbe9c0-443c-4551-93f9-1b50010ff9f0" containerID="ae9e88d90a6c9997803e33594a2c4fdcc640016ef80923217df8ff6fcc6fa0db" exitCode=0
Nov 03 20:39:11.933049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.932887    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"
Nov 03 20:39:11.933049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.932889    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58" event={"ID":"bfcbe9c0-443c-4551-93f9-1b50010ff9f0","Type":"ContainerDied","Data":"ae9e88d90a6c9997803e33594a2c4fdcc640016ef80923217df8ff6fcc6fa0db"}
Nov 03 20:39:11.933157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.933053    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58" event={"ID":"bfcbe9c0-443c-4551-93f9-1b50010ff9f0","Type":"ContainerDied","Data":"a820025b1428133936e84bce9467b6bd745304b5b081afd5ac7959c5d85ffc35"}
Nov 03 20:39:11.933157 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.933087    5659 scope.go:122] "RemoveContainer" containerID="ae9e88d90a6c9997803e33594a2c4fdcc640016ef80923217df8ff6fcc6fa0db"
Nov 03 20:39:11.935336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.935304    5659 generic.go:356] "Generic (PLEG): container finished" podID="a7356100-2a57-4289-b38f-133d56fa44d1" containerID="a61626e49efa7334c7e4ab289ee564a4b49614c796efd0a4f04bd466ef945502" exitCode=0
Nov 03 20:39:11.935466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.935353    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd" event={"ID":"a7356100-2a57-4289-b38f-133d56fa44d1","Type":"ContainerDied","Data":"a61626e49efa7334c7e4ab289ee564a4b49614c796efd0a4f04bd466ef945502"}
Nov 03 20:39:11.935466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.935409    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd" event={"ID":"a7356100-2a57-4289-b38f-133d56fa44d1","Type":"ContainerDied","Data":"5caccd540619dcf81cadf3473052e6507eb1506af1b0b57b9583c9cd7254a7d1"}
Nov 03 20:39:11.935466 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.935362    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"
Nov 03 20:39:11.946115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.946096    5659 scope.go:122] "RemoveContainer" containerID="ae9e88d90a6c9997803e33594a2c4fdcc640016ef80923217df8ff6fcc6fa0db"
Nov 03 20:39:11.946387 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:11.946363    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9e88d90a6c9997803e33594a2c4fdcc640016ef80923217df8ff6fcc6fa0db\": container with ID starting with ae9e88d90a6c9997803e33594a2c4fdcc640016ef80923217df8ff6fcc6fa0db not found: ID does not exist" containerID="ae9e88d90a6c9997803e33594a2c4fdcc640016ef80923217df8ff6fcc6fa0db"
Nov 03 20:39:11.946473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.946395    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9e88d90a6c9997803e33594a2c4fdcc640016ef80923217df8ff6fcc6fa0db"} err="failed to get container status \"ae9e88d90a6c9997803e33594a2c4fdcc640016ef80923217df8ff6fcc6fa0db\": rpc error: code = NotFound desc = could not find container \"ae9e88d90a6c9997803e33594a2c4fdcc640016ef80923217df8ff6fcc6fa0db\": container with ID starting with ae9e88d90a6c9997803e33594a2c4fdcc640016ef80923217df8ff6fcc6fa0db not found: ID does not exist"
Nov 03 20:39:11.946473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.946413    5659 scope.go:122] "RemoveContainer" containerID="a61626e49efa7334c7e4ab289ee564a4b49614c796efd0a4f04bd466ef945502"
Nov 03 20:39:11.962654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.962626    5659 scope.go:122] "RemoveContainer" containerID="a61626e49efa7334c7e4ab289ee564a4b49614c796efd0a4f04bd466ef945502"
Nov 03 20:39:11.963065 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:11.963042    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61626e49efa7334c7e4ab289ee564a4b49614c796efd0a4f04bd466ef945502\": container with ID starting with a61626e49efa7334c7e4ab289ee564a4b49614c796efd0a4f04bd466ef945502 not found: ID does not exist" containerID="a61626e49efa7334c7e4ab289ee564a4b49614c796efd0a4f04bd466ef945502"
Nov 03 20:39:11.963186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.963076    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61626e49efa7334c7e4ab289ee564a4b49614c796efd0a4f04bd466ef945502"} err="failed to get container status \"a61626e49efa7334c7e4ab289ee564a4b49614c796efd0a4f04bd466ef945502\": rpc error: code = NotFound desc = could not find container \"a61626e49efa7334c7e4ab289ee564a4b49614c796efd0a4f04bd466ef945502\": container with ID starting with a61626e49efa7334c7e4ab289ee564a4b49614c796efd0a4f04bd466ef945502 not found: ID does not exist"
Nov 03 20:39:11.973443 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.973416    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" path="/var/lib/kubelet/pods/72c08e50-44dd-4bf9-ae14-f24e2672e4c5/volumes"
Nov 03 20:39:11.973924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.973898    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"]
Nov 03 20:39:11.973978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.973937    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-33e267-867b6cfb59-lsm58"]
Nov 03 20:39:11.985257 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.985226    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"]
Nov 03 20:39:11.988804 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:11.988770    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-33e267-7fb8df65fb-6ctzd"]
Nov 03 20:39:12.165503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:12.157339    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:12.269536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:12.267292    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"]
Nov 03 20:39:12.359989 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:12.359956    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:12.495159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:12.495128    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:12.948317 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:12.948270    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:13.245799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:13.245762    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:13.380281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:13.380241    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:13.849767 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:13.849731    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:13.971100 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:13.971064    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7356100-2a57-4289-b38f-133d56fa44d1" path="/var/lib/kubelet/pods/a7356100-2a57-4289-b38f-133d56fa44d1/volumes"
Nov 03 20:39:13.971641 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:13.971619    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfcbe9c0-443c-4551-93f9-1b50010ff9f0" path="/var/lib/kubelet/pods/bfcbe9c0-443c-4551-93f9-1b50010ff9f0/volumes"
Nov 03 20:39:14.196956 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:14.196873    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:14.340506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:14.337385    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:14.926106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:14.926066    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:15.153505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:15.150935    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:15.219963 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:15.219929    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:39:16.028522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:16.027911    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:16.649421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:16.649386    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:16.779773 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:16.779732    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:16.971507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:16.965119    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:17.105708 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:17.105670    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:17.267962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:17.267853    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq" podUID="3079f0fd-1225-42ab-8151-4062129bcca0" containerName="payload-processing" containerID="cri-o://4957526e2ac76067ecfc42871eb8fa8794ac687c8ba7601061721a231ed6e926" gracePeriod=25
Nov 03 20:39:17.631641 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:17.631558    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:17.987998 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:17.987960    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:18.364101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:18.364001    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:18.782960 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:18.782894    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:19.439557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:19.439470    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:20.140616 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:20.140529    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:20.694174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:20.694139    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:20.836855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:20.836818    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:21.051295 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:21.051248    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:21.321527 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:21.321406    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:21.599742 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:21.599659    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"]
Nov 03 20:39:21.600187 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:21.600157    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w" podUID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerName="istio-proxy" containerID="cri-o://53685b48da32a7f271eb55f633edeb36f59e20e79499279cbb0766d9cc11ebf2" gracePeriod=30
Nov 03 20:39:21.674727 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:21.674695    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:22.137063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:22.137027    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:22.270160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:22.270123    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:22.401576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:22.401227    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:23.002801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:23.002770    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:23.579057 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:23.579022    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:23.913656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:23.913571    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:24.531515 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:24.531467    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:24.717792 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:24.717755    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:25.308521 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:25.308462    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq" podUID="3079f0fd-1225-42ab-8151-4062129bcca0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:39:25.396088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:25.396047    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:25.911044 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:25.911010    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:26.147127 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:26.147094    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:26.864289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:26.864249    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:26.956297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:26.956260    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:39:27.074240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.074200    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-podinfo\") pod \"8596f4f0-3ef7-446c-8374-57bcf455301c\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") "
Nov 03 20:39:27.074423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.074262    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-credential-socket\") pod \"8596f4f0-3ef7-446c-8374-57bcf455301c\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") "
Nov 03 20:39:27.074423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.074372    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-socket\") pod \"8596f4f0-3ef7-446c-8374-57bcf455301c\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") "
Nov 03 20:39:27.074423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.074410    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8596f4f0-3ef7-446c-8374-57bcf455301c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8596f4f0-3ef7-446c-8374-57bcf455301c-istiod-ca-cert\") pod \"8596f4f0-3ef7-446c-8374-57bcf455301c\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") "
Nov 03 20:39:27.074631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.074472    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-kube-api-access-2gqvq\" (UniqueName: \"kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-kube-api-access-2gqvq\") pod \"8596f4f0-3ef7-446c-8374-57bcf455301c\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") "
Nov 03 20:39:27.074631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.074519    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-credential-socket" pod "8596f4f0-3ef7-446c-8374-57bcf455301c" (UID: "8596f4f0-3ef7-446c-8374-57bcf455301c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:27.074748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.074643    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-envoy\") pod \"8596f4f0-3ef7-446c-8374-57bcf455301c\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") "
Nov 03 20:39:27.074748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.074678    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-certs\") pod \"8596f4f0-3ef7-446c-8374-57bcf455301c\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") "
Nov 03 20:39:27.074748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.074688    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-socket" pod "8596f4f0-3ef7-446c-8374-57bcf455301c" (UID: "8596f4f0-3ef7-446c-8374-57bcf455301c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:27.074748 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.074733    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-data\") pod \"8596f4f0-3ef7-446c-8374-57bcf455301c\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") "
Nov 03 20:39:27.074981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.074766    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-token\" (UniqueName: \"kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-token\") pod \"8596f4f0-3ef7-446c-8374-57bcf455301c\" (UID: \"8596f4f0-3ef7-446c-8374-57bcf455301c\") "
Nov 03 20:39:27.074981 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.074826    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8596f4f0-3ef7-446c-8374-57bcf455301c-istiod-ca-cert" pod "8596f4f0-3ef7-446c-8374-57bcf455301c" (UID: "8596f4f0-3ef7-446c-8374-57bcf455301c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:27.075099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.074996    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-certs" pod "8596f4f0-3ef7-446c-8374-57bcf455301c" (UID: "8596f4f0-3ef7-446c-8374-57bcf455301c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:27.075099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.075090    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-data" pod "8596f4f0-3ef7-446c-8374-57bcf455301c" (UID: "8596f4f0-3ef7-446c-8374-57bcf455301c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:27.075435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.075414    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8596f4f0-3ef7-446c-8374-57bcf455301c-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:27.075435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.075434    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:27.075662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.075443    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:27.075662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.075453    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:27.075662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.075461    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:27.077179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.077150    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-podinfo" pod "8596f4f0-3ef7-446c-8374-57bcf455301c" (UID: "8596f4f0-3ef7-446c-8374-57bcf455301c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:39:27.077275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.077183    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-kube-api-access-2gqvq" pod "8596f4f0-3ef7-446c-8374-57bcf455301c" (UID: "8596f4f0-3ef7-446c-8374-57bcf455301c"). InnerVolumeSpecName "kube-api-access-2gqvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:27.077275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.077215    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-token" pod "8596f4f0-3ef7-446c-8374-57bcf455301c" (UID: "8596f4f0-3ef7-446c-8374-57bcf455301c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:27.077275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.077218    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-envoy" pod "8596f4f0-3ef7-446c-8374-57bcf455301c" (UID: "8596f4f0-3ef7-446c-8374-57bcf455301c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:27.112217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.112181    5659 generic.go:356] "Generic (PLEG): container finished" podID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerID="53685b48da32a7f271eb55f633edeb36f59e20e79499279cbb0766d9cc11ebf2" exitCode=0
Nov 03 20:39:27.112379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.112214    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w" event={"ID":"8596f4f0-3ef7-446c-8374-57bcf455301c","Type":"ContainerDied","Data":"53685b48da32a7f271eb55f633edeb36f59e20e79499279cbb0766d9cc11ebf2"}
Nov 03 20:39:27.112379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.112240    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"
Nov 03 20:39:27.112379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.112277    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w" event={"ID":"8596f4f0-3ef7-446c-8374-57bcf455301c","Type":"ContainerDied","Data":"d586b54f0ffe303b888360c1e0db5570da5d234df4bdce5768cd3de3e92a24e4"}
Nov 03 20:39:27.112379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.112300    5659 scope.go:122] "RemoveContainer" containerID="53685b48da32a7f271eb55f633edeb36f59e20e79499279cbb0766d9cc11ebf2"
Nov 03 20:39:27.124852 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.124835    5659 scope.go:122] "RemoveContainer" containerID="4a5260a0b8cc18c20c0f6d1dd154cbc65cfd5b01e5c7c4b8497c326a357089f5"
Nov 03 20:39:27.136418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.136390    5659 scope.go:122] "RemoveContainer" containerID="53685b48da32a7f271eb55f633edeb36f59e20e79499279cbb0766d9cc11ebf2"
Nov 03 20:39:27.136795 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:27.136767    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53685b48da32a7f271eb55f633edeb36f59e20e79499279cbb0766d9cc11ebf2\": container with ID starting with 53685b48da32a7f271eb55f633edeb36f59e20e79499279cbb0766d9cc11ebf2 not found: ID does not exist" containerID="53685b48da32a7f271eb55f633edeb36f59e20e79499279cbb0766d9cc11ebf2"
Nov 03 20:39:27.136882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.136806    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53685b48da32a7f271eb55f633edeb36f59e20e79499279cbb0766d9cc11ebf2"} err="failed to get container status \"53685b48da32a7f271eb55f633edeb36f59e20e79499279cbb0766d9cc11ebf2\": rpc error: code = NotFound desc = could not find container \"53685b48da32a7f271eb55f633edeb36f59e20e79499279cbb0766d9cc11ebf2\": container with ID starting with 53685b48da32a7f271eb55f633edeb36f59e20e79499279cbb0766d9cc11ebf2 not found: ID does not exist"
Nov 03 20:39:27.136882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.136831    5659 scope.go:122] "RemoveContainer" containerID="4a5260a0b8cc18c20c0f6d1dd154cbc65cfd5b01e5c7c4b8497c326a357089f5"
Nov 03 20:39:27.137169 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:27.137141    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5260a0b8cc18c20c0f6d1dd154cbc65cfd5b01e5c7c4b8497c326a357089f5\": container with ID starting with 4a5260a0b8cc18c20c0f6d1dd154cbc65cfd5b01e5c7c4b8497c326a357089f5 not found: ID does not exist" containerID="4a5260a0b8cc18c20c0f6d1dd154cbc65cfd5b01e5c7c4b8497c326a357089f5"
Nov 03 20:39:27.137233 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.137179    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5260a0b8cc18c20c0f6d1dd154cbc65cfd5b01e5c7c4b8497c326a357089f5"} err="failed to get container status \"4a5260a0b8cc18c20c0f6d1dd154cbc65cfd5b01e5c7c4b8497c326a357089f5\": rpc error: code = NotFound desc = could not find container \"4a5260a0b8cc18c20c0f6d1dd154cbc65cfd5b01e5c7c4b8497c326a357089f5\": container with ID starting with 4a5260a0b8cc18c20c0f6d1dd154cbc65cfd5b01e5c7c4b8497c326a357089f5 not found: ID does not exist"
Nov 03 20:39:27.140702 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.140679    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"]
Nov 03 20:39:27.144148 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.144128    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-836265-openshift-default-67b9f59555-b4k5w"]
Nov 03 20:39:27.177324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.177233    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gqvq\" (UniqueName: \"kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-kube-api-access-2gqvq\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:27.177324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.177276    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:27.177324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.177295    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:27.177324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.177310    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8596f4f0-3ef7-446c-8374-57bcf455301c-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:27.652600 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.652561    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:27.845329 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.845288    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"]
Nov 03 20:39:27.864253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.864212    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"]
Nov 03 20:39:27.949293 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.949206    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:27.971332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:27.971302    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8596f4f0-3ef7-446c-8374-57bcf455301c" path="/var/lib/kubelet/pods/8596f4f0-3ef7-446c-8374-57bcf455301c/volumes"
Nov 03 20:39:28.104509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:28.096118    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:28.705153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:28.705109    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:29.233332 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:29.233300    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:29.362802 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:29.362768    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:29.564666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:29.563002    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:29.703162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:29.703129    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:29.832896 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:29.832802    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:30.060670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:30.059886    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"]
Nov 03 20:39:30.060670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:30.060137    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c" podUID="aba237d4-70b3-4347-86b6-462fc504fe3d" containerName="istio-proxy" containerID="cri-o://697a722756df43548d3fc79ede2b5977cae0a2e6906a2a790ac5a1c955d4bf77" gracePeriod=30
Nov 03 20:39:30.145092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:30.144983    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:30.529729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:30.529554    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:30.913454 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:30.913348    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:31.277701 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:31.277666    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:31.441524 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:31.440619    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:31.583130 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:31.583048    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:32.211712 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:32.211673    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:32.356921 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:32.356888    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:32.513966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:32.513876    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"]
Nov 03 20:39:32.514162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:32.514126    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f" podUID="d468b03c-81e8-44a3-948e-ed1e86de9aca" containerName="istio-proxy" containerID="cri-o://2b1977099f6cc32e2b75fcf97d7d3929e9bbed5fffe3c68bcbcf1db1bebc24a7" gracePeriod=30
Nov 03 20:39:32.567971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:32.567933    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:32.856460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:32.856402    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc" podUID="376c36bb-8e35-46d3-9ced-b2a66f7370f7" containerName="payload-processing" containerID="cri-o://35887100fca9ea6d3ba2319a22b918fab53f2b0f873cb23f2f12cbc4027e3d4f" gracePeriod=25
Nov 03 20:39:32.865656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:32.865610    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx" podUID="fbf0a278-c34b-484d-9b5e-11243d3c2c63" containerName="payload-pre-processing" containerID="cri-o://c239731ac9c2e45cd9c00b0f6862a4129ed9c55588a868ccf1d5eab48f97e263" gracePeriod=25
Nov 03 20:39:32.893641 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:32.893613    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:33.215824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:33.215762    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:33.364498 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:33.364462    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:33.946780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:33.946743    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:34.560441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:34.560400    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc" podUID="376c36bb-8e35-46d3-9ced-b2a66f7370f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:39:34.783873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:34.780776    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:35.251788 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.251749    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:35.276181 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.276153    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:39:35.308744 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.308708    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq" podUID="3079f0fd-1225-42ab-8151-4062129bcca0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:39:35.352007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.349530    5659 generic.go:356] "Generic (PLEG): container finished" podID="aba237d4-70b3-4347-86b6-462fc504fe3d" containerID="697a722756df43548d3fc79ede2b5977cae0a2e6906a2a790ac5a1c955d4bf77" exitCode=0
Nov 03 20:39:35.352007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.349627    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c" event={"ID":"aba237d4-70b3-4347-86b6-462fc504fe3d","Type":"ContainerDied","Data":"697a722756df43548d3fc79ede2b5977cae0a2e6906a2a790ac5a1c955d4bf77"}
Nov 03 20:39:35.352007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.349684    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c" event={"ID":"aba237d4-70b3-4347-86b6-462fc504fe3d","Type":"ContainerDied","Data":"a6a9ea2791e953e49792b0e7431c803498ba7783c985d481dde3175f9de1cba9"}
Nov 03 20:39:35.352007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.349716    5659 scope.go:122] "RemoveContainer" containerID="697a722756df43548d3fc79ede2b5977cae0a2e6906a2a790ac5a1c955d4bf77"
Nov 03 20:39:35.352007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.349888    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"
Nov 03 20:39:35.367684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.367627    5659 scope.go:122] "RemoveContainer" containerID="697a722756df43548d3fc79ede2b5977cae0a2e6906a2a790ac5a1c955d4bf77"
Nov 03 20:39:35.368005 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:35.367982    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697a722756df43548d3fc79ede2b5977cae0a2e6906a2a790ac5a1c955d4bf77\": container with ID starting with 697a722756df43548d3fc79ede2b5977cae0a2e6906a2a790ac5a1c955d4bf77 not found: ID does not exist" containerID="697a722756df43548d3fc79ede2b5977cae0a2e6906a2a790ac5a1c955d4bf77"
Nov 03 20:39:35.368101 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.368017    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697a722756df43548d3fc79ede2b5977cae0a2e6906a2a790ac5a1c955d4bf77"} err="failed to get container status \"697a722756df43548d3fc79ede2b5977cae0a2e6906a2a790ac5a1c955d4bf77\": rpc error: code = NotFound desc = could not find container \"697a722756df43548d3fc79ede2b5977cae0a2e6906a2a790ac5a1c955d4bf77\": container with ID starting with 697a722756df43548d3fc79ede2b5977cae0a2e6906a2a790ac5a1c955d4bf77 not found: ID does not exist"
Nov 03 20:39:35.379530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.379459    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-podinfo\") pod \"aba237d4-70b3-4347-86b6-462fc504fe3d\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") "
Nov 03 20:39:35.379712 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.379609    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-credential-socket\") pod \"aba237d4-70b3-4347-86b6-462fc504fe3d\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") "
Nov 03 20:39:35.379786 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.379757    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-token\" (UniqueName: \"kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-token\") pod \"aba237d4-70b3-4347-86b6-462fc504fe3d\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") "
Nov 03 20:39:35.379846 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.379827    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-data\") pod \"aba237d4-70b3-4347-86b6-462fc504fe3d\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") "
Nov 03 20:39:35.379903 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.379875    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-envoy\") pod \"aba237d4-70b3-4347-86b6-462fc504fe3d\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") "
Nov 03 20:39:35.379966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.379911    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-socket\") pod \"aba237d4-70b3-4347-86b6-462fc504fe3d\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") "
Nov 03 20:39:35.380028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.379970    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-kube-api-access-cb5sj\" (UniqueName: \"kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-kube-api-access-cb5sj\") pod \"aba237d4-70b3-4347-86b6-462fc504fe3d\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") "
Nov 03 20:39:35.380028 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.380004    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba237d4-70b3-4347-86b6-462fc504fe3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba237d4-70b3-4347-86b6-462fc504fe3d-istiod-ca-cert\") pod \"aba237d4-70b3-4347-86b6-462fc504fe3d\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") "
Nov 03 20:39:35.380134 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.380029    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-certs\") pod \"aba237d4-70b3-4347-86b6-462fc504fe3d\" (UID: \"aba237d4-70b3-4347-86b6-462fc504fe3d\") "
Nov 03 20:39:35.380976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.380918    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-credential-socket" pod "aba237d4-70b3-4347-86b6-462fc504fe3d" (UID: "aba237d4-70b3-4347-86b6-462fc504fe3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:35.381094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.381015    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-socket" pod "aba237d4-70b3-4347-86b6-462fc504fe3d" (UID: "aba237d4-70b3-4347-86b6-462fc504fe3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:35.381402 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.381361    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-data" pod "aba237d4-70b3-4347-86b6-462fc504fe3d" (UID: "aba237d4-70b3-4347-86b6-462fc504fe3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:35.381402 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.381365    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-certs" pod "aba237d4-70b3-4347-86b6-462fc504fe3d" (UID: "aba237d4-70b3-4347-86b6-462fc504fe3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:35.381698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.381673    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba237d4-70b3-4347-86b6-462fc504fe3d-istiod-ca-cert" pod "aba237d4-70b3-4347-86b6-462fc504fe3d" (UID: "aba237d4-70b3-4347-86b6-462fc504fe3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:35.384172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.383875    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-token" pod "aba237d4-70b3-4347-86b6-462fc504fe3d" (UID: "aba237d4-70b3-4347-86b6-462fc504fe3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:35.384286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.384237    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-podinfo" pod "aba237d4-70b3-4347-86b6-462fc504fe3d" (UID: "aba237d4-70b3-4347-86b6-462fc504fe3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:39:35.384460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.384426    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-envoy" pod "aba237d4-70b3-4347-86b6-462fc504fe3d" (UID: "aba237d4-70b3-4347-86b6-462fc504fe3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:35.385952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.385929    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-kube-api-access-cb5sj" pod "aba237d4-70b3-4347-86b6-462fc504fe3d" (UID: "aba237d4-70b3-4347-86b6-462fc504fe3d"). InnerVolumeSpecName "kube-api-access-cb5sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:35.403553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.403517    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:35.500509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.488736    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.500509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.488792    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.500509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.488808    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.500509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.488819    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.500509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.488829    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb5sj\" (UniqueName: \"kubernetes.io/projected/aba237d4-70b3-4347-86b6-462fc504fe3d-kube-api-access-cb5sj\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.500509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.488851    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba237d4-70b3-4347-86b6-462fc504fe3d-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.500509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.488865    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.500509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.488877    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba237d4-70b3-4347-86b6-462fc504fe3d-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.500509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.488886    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba237d4-70b3-4347-86b6-462fc504fe3d-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:35.544223 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.544177    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:35.600382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.600340    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx" podUID="fbf0a278-c34b-484d-9b5e-11243d3c2c63" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:39:35.680452 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.680414    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"]
Nov 03 20:39:35.683205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.683181    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4c2a28-openshift-default-7b6689f6b9-w282c"]
Nov 03 20:39:35.736507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:35.729346    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:36.010289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:36.010227    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba237d4-70b3-4347-86b6-462fc504fe3d" path="/var/lib/kubelet/pods/aba237d4-70b3-4347-86b6-462fc504fe3d/volumes"
Nov 03 20:39:36.027044 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:36.027000    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"]
Nov 03 20:39:36.027648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:36.027613    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh" podUID="6c85f57d-183f-408c-a450-6d6ad119a7db" containerName="istio-proxy" containerID="cri-o://887b1c1799d538caba2c3e7fe42b7234a959cb83671d40f8a5b5ea942aef2b09" gracePeriod=30
Nov 03 20:39:36.075367 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:36.075323    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:36.415563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:36.412183    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:36.546840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:36.546801    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:37.006305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.006245    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:37.133745 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.133710    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:37.523394 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.523363    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:37.690214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.689862    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"
Nov 03 20:39:37.791851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.791080    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3079f0fd-1225-42ab-8151-4062129bcca0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3079f0fd-1225-42ab-8151-4062129bcca0-plugins-config-volume\") pod \"3079f0fd-1225-42ab-8151-4062129bcca0\" (UID: \"3079f0fd-1225-42ab-8151-4062129bcca0\") "
Nov 03 20:39:37.791851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.791137    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3079f0fd-1225-42ab-8151-4062129bcca0-kube-api-access-l4dzm\" (UniqueName: \"kubernetes.io/projected/3079f0fd-1225-42ab-8151-4062129bcca0-kube-api-access-l4dzm\") pod \"3079f0fd-1225-42ab-8151-4062129bcca0\" (UID: \"3079f0fd-1225-42ab-8151-4062129bcca0\") "
Nov 03 20:39:37.791851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.791560    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3079f0fd-1225-42ab-8151-4062129bcca0-plugins-config-volume" pod "3079f0fd-1225-42ab-8151-4062129bcca0" (UID: "3079f0fd-1225-42ab-8151-4062129bcca0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:37.791851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.791807    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3079f0fd-1225-42ab-8151-4062129bcca0-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:37.793961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.793918    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3079f0fd-1225-42ab-8151-4062129bcca0-kube-api-access-l4dzm" pod "3079f0fd-1225-42ab-8151-4062129bcca0" (UID: "3079f0fd-1225-42ab-8151-4062129bcca0"). InnerVolumeSpecName "kube-api-access-l4dzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:37.861945 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.861913    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:39:37.893494 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.893447    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4dzm\" (UniqueName: \"kubernetes.io/projected/3079f0fd-1225-42ab-8151-4062129bcca0-kube-api-access-l4dzm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:37.928799 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.928716    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:37.994291 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.994261    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-credential-socket\") pod \"d468b03c-81e8-44a3-948e-ed1e86de9aca\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") "
Nov 03 20:39:37.994436 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.994315    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-token\" (UniqueName: \"kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-token\") pod \"d468b03c-81e8-44a3-948e-ed1e86de9aca\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") "
Nov 03 20:39:37.994436 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.994406    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-socket\") pod \"d468b03c-81e8-44a3-948e-ed1e86de9aca\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") "
Nov 03 20:39:37.994552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.994445    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-data\") pod \"d468b03c-81e8-44a3-948e-ed1e86de9aca\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") "
Nov 03 20:39:37.994607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.994558    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-envoy\") pod \"d468b03c-81e8-44a3-948e-ed1e86de9aca\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") "
Nov 03 20:39:37.994607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.994578    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-credential-socket" pod "d468b03c-81e8-44a3-948e-ed1e86de9aca" (UID: "d468b03c-81e8-44a3-948e-ed1e86de9aca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:37.994607 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.994599    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d468b03c-81e8-44a3-948e-ed1e86de9aca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d468b03c-81e8-44a3-948e-ed1e86de9aca-istiod-ca-cert\") pod \"d468b03c-81e8-44a3-948e-ed1e86de9aca\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") "
Nov 03 20:39:37.994870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.994641    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-certs\") pod \"d468b03c-81e8-44a3-948e-ed1e86de9aca\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") "
Nov 03 20:39:37.994870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.994711    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-socket" pod "d468b03c-81e8-44a3-948e-ed1e86de9aca" (UID: "d468b03c-81e8-44a3-948e-ed1e86de9aca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:37.994870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.994718    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-kube-api-access-8l97t\" (UniqueName: \"kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-kube-api-access-8l97t\") pod \"d468b03c-81e8-44a3-948e-ed1e86de9aca\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") "
Nov 03 20:39:37.994870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.994753    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-data" pod "d468b03c-81e8-44a3-948e-ed1e86de9aca" (UID: "d468b03c-81e8-44a3-948e-ed1e86de9aca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:37.994870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.994792    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-podinfo\") pod \"d468b03c-81e8-44a3-948e-ed1e86de9aca\" (UID: \"d468b03c-81e8-44a3-948e-ed1e86de9aca\") "
Nov 03 20:39:37.995159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.994997    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-certs" pod "d468b03c-81e8-44a3-948e-ed1e86de9aca" (UID: "d468b03c-81e8-44a3-948e-ed1e86de9aca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:37.995159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.995065    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d468b03c-81e8-44a3-948e-ed1e86de9aca-istiod-ca-cert" pod "d468b03c-81e8-44a3-948e-ed1e86de9aca" (UID: "d468b03c-81e8-44a3-948e-ed1e86de9aca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:37.995398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.995375    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:37.995560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.995401    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:37.995560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.995417    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:37.995560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.995429    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d468b03c-81e8-44a3-948e-ed1e86de9aca-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:37.995560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.995438    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:37.996750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.996730    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-token" pod "d468b03c-81e8-44a3-948e-ed1e86de9aca" (UID: "d468b03c-81e8-44a3-948e-ed1e86de9aca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:37.996997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.996984    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-podinfo" pod "d468b03c-81e8-44a3-948e-ed1e86de9aca" (UID: "d468b03c-81e8-44a3-948e-ed1e86de9aca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:39:37.997238 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.997218    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-envoy" pod "d468b03c-81e8-44a3-948e-ed1e86de9aca" (UID: "d468b03c-81e8-44a3-948e-ed1e86de9aca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:37.997308 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:37.997262    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-kube-api-access-8l97t" pod "d468b03c-81e8-44a3-948e-ed1e86de9aca" (UID: "d468b03c-81e8-44a3-948e-ed1e86de9aca"). InnerVolumeSpecName "kube-api-access-8l97t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:38.097088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.096851    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l97t\" (UniqueName: \"kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-kube-api-access-8l97t\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.097088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.096889    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.097088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.096905    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.097088 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.096920    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d468b03c-81e8-44a3-948e-ed1e86de9aca-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.406564 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.405607    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:38.410698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.409818    5659 generic.go:356] "Generic (PLEG): container finished" podID="d468b03c-81e8-44a3-948e-ed1e86de9aca" containerID="2b1977099f6cc32e2b75fcf97d7d3929e9bbed5fffe3c68bcbcf1db1bebc24a7" exitCode=0
Nov 03 20:39:38.410698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.410096    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f" event={"ID":"d468b03c-81e8-44a3-948e-ed1e86de9aca","Type":"ContainerDied","Data":"2b1977099f6cc32e2b75fcf97d7d3929e9bbed5fffe3c68bcbcf1db1bebc24a7"}
Nov 03 20:39:38.410698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.410157    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f" event={"ID":"d468b03c-81e8-44a3-948e-ed1e86de9aca","Type":"ContainerDied","Data":"05c45c422883690b31e604243a24d22b2efeeac63514417bc516be600adc4a2c"}
Nov 03 20:39:38.410698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.410187    5659 scope.go:122] "RemoveContainer" containerID="2b1977099f6cc32e2b75fcf97d7d3929e9bbed5fffe3c68bcbcf1db1bebc24a7"
Nov 03 20:39:38.410698 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.410326    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"
Nov 03 20:39:38.418560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.418369    5659 generic.go:356] "Generic (PLEG): container finished" podID="3079f0fd-1225-42ab-8151-4062129bcca0" containerID="4957526e2ac76067ecfc42871eb8fa8794ac687c8ba7601061721a231ed6e926" exitCode=0
Nov 03 20:39:38.418560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.418470    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq" event={"ID":"3079f0fd-1225-42ab-8151-4062129bcca0","Type":"ContainerDied","Data":"4957526e2ac76067ecfc42871eb8fa8794ac687c8ba7601061721a231ed6e926"}
Nov 03 20:39:38.418560 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.418538    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq" event={"ID":"3079f0fd-1225-42ab-8151-4062129bcca0","Type":"ContainerDied","Data":"d378adaa513bcb6fc1faaf8394801f69770d5fecc9b88844878d1726d5e867c2"}
Nov 03 20:39:38.418759 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.418590    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"
Nov 03 20:39:38.428963 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.428943    5659 scope.go:122] "RemoveContainer" containerID="2b1977099f6cc32e2b75fcf97d7d3929e9bbed5fffe3c68bcbcf1db1bebc24a7"
Nov 03 20:39:38.429269 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:38.429248    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1977099f6cc32e2b75fcf97d7d3929e9bbed5fffe3c68bcbcf1db1bebc24a7\": container with ID starting with 2b1977099f6cc32e2b75fcf97d7d3929e9bbed5fffe3c68bcbcf1db1bebc24a7 not found: ID does not exist" containerID="2b1977099f6cc32e2b75fcf97d7d3929e9bbed5fffe3c68bcbcf1db1bebc24a7"
Nov 03 20:39:38.429316 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.429280    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1977099f6cc32e2b75fcf97d7d3929e9bbed5fffe3c68bcbcf1db1bebc24a7"} err="failed to get container status \"2b1977099f6cc32e2b75fcf97d7d3929e9bbed5fffe3c68bcbcf1db1bebc24a7\": rpc error: code = NotFound desc = could not find container \"2b1977099f6cc32e2b75fcf97d7d3929e9bbed5fffe3c68bcbcf1db1bebc24a7\": container with ID starting with 2b1977099f6cc32e2b75fcf97d7d3929e9bbed5fffe3c68bcbcf1db1bebc24a7 not found: ID does not exist"
Nov 03 20:39:38.429316 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.429296    5659 scope.go:122] "RemoveContainer" containerID="4957526e2ac76067ecfc42871eb8fa8794ac687c8ba7601061721a231ed6e926"
Nov 03 20:39:38.440674 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.440641    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"]
Nov 03 20:39:38.442367 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.442343    5659 scope.go:122] "RemoveContainer" containerID="4957526e2ac76067ecfc42871eb8fa8794ac687c8ba7601061721a231ed6e926"
Nov 03 20:39:38.442733 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:38.442709    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4957526e2ac76067ecfc42871eb8fa8794ac687c8ba7601061721a231ed6e926\": container with ID starting with 4957526e2ac76067ecfc42871eb8fa8794ac687c8ba7601061721a231ed6e926 not found: ID does not exist" containerID="4957526e2ac76067ecfc42871eb8fa8794ac687c8ba7601061721a231ed6e926"
Nov 03 20:39:38.442862 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.442743    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4957526e2ac76067ecfc42871eb8fa8794ac687c8ba7601061721a231ed6e926"} err="failed to get container status \"4957526e2ac76067ecfc42871eb8fa8794ac687c8ba7601061721a231ed6e926\": rpc error: code = NotFound desc = could not find container \"4957526e2ac76067ecfc42871eb8fa8794ac687c8ba7601061721a231ed6e926\": container with ID starting with 4957526e2ac76067ecfc42871eb8fa8794ac687c8ba7601061721a231ed6e926 not found: ID does not exist"
Nov 03 20:39:38.444534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.444468    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-836265-7cd9bb4d68-7dtlq"]
Nov 03 20:39:38.459120 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.459063    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"]
Nov 03 20:39:38.463498 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:38.463458    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-nzb4f"]
Nov 03 20:39:39.113515 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:39.113471    5659 token_manager.go:124] "Couldn't update token" err="serviceaccounts \"default\" is forbidden: unable to create new content in namespace e2e-models-e2e-worker-w2-2b5703 because it is being terminated" cacheKey="\"default\"/\"e2e-models-e2e-worker-w2-2b5703\"/[]string(nil)/3607/v1.BoundObjectReference{Kind:\"Pod\", APIVersion:\"v1\", Name:\"distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t\", UID:\"f2d9f7b7-51af-4be2-b959-3143ebc16f58\"}/\"\""
Nov 03 20:39:39.239812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:39.239777    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:39.294595 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:39.294553    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"]
Nov 03 20:39:39.295854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:39.295829    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"]
Nov 03 20:39:39.392082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:39.388774    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:39.545558 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:39.541559    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:39.976274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:39.976236    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3079f0fd-1225-42ab-8151-4062129bcca0" path="/var/lib/kubelet/pods/3079f0fd-1225-42ab-8151-4062129bcca0/volumes"
Nov 03 20:39:39.976779 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:39.976761    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d468b03c-81e8-44a3-948e-ed1e86de9aca" path="/var/lib/kubelet/pods/d468b03c-81e8-44a3-948e-ed1e86de9aca/volumes"
Nov 03 20:39:40.153552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:40.153516    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:40.212793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:40.212745    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:39:40.212987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:40.212824    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh" podUID="6c85f57d-183f-408c-a450-6d6ad119a7db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:39:40.289787 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:40.289674    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:40.881820 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:40.862459    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:41.342857 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.342821    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:39:41.406928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.406845    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:41.444023 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.443982    5659 generic.go:356] "Generic (PLEG): container finished" podID="6c85f57d-183f-408c-a450-6d6ad119a7db" containerID="887b1c1799d538caba2c3e7fe42b7234a959cb83671d40f8a5b5ea942aef2b09" exitCode=0
Nov 03 20:39:41.444221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.444048    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"
Nov 03 20:39:41.444221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.444068    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh" event={"ID":"6c85f57d-183f-408c-a450-6d6ad119a7db","Type":"ContainerDied","Data":"887b1c1799d538caba2c3e7fe42b7234a959cb83671d40f8a5b5ea942aef2b09"}
Nov 03 20:39:41.444221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.444150    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh" event={"ID":"6c85f57d-183f-408c-a450-6d6ad119a7db","Type":"ContainerDied","Data":"b3a9f156abb0de4ebfb99c27ef9c355934960892e801641f5ba94699718b52c6"}
Nov 03 20:39:41.444221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.444174    5659 scope.go:122] "RemoveContainer" containerID="887b1c1799d538caba2c3e7fe42b7234a959cb83671d40f8a5b5ea942aef2b09"
Nov 03 20:39:41.456798 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.456773    5659 scope.go:122] "RemoveContainer" containerID="887b1c1799d538caba2c3e7fe42b7234a959cb83671d40f8a5b5ea942aef2b09"
Nov 03 20:39:41.457164 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:41.457140    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887b1c1799d538caba2c3e7fe42b7234a959cb83671d40f8a5b5ea942aef2b09\": container with ID starting with 887b1c1799d538caba2c3e7fe42b7234a959cb83671d40f8a5b5ea942aef2b09 not found: ID does not exist" containerID="887b1c1799d538caba2c3e7fe42b7234a959cb83671d40f8a5b5ea942aef2b09"
Nov 03 20:39:41.457270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.457172    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887b1c1799d538caba2c3e7fe42b7234a959cb83671d40f8a5b5ea942aef2b09"} err="failed to get container status \"887b1c1799d538caba2c3e7fe42b7234a959cb83671d40f8a5b5ea942aef2b09\": rpc error: code = NotFound desc = could not find container \"887b1c1799d538caba2c3e7fe42b7234a959cb83671d40f8a5b5ea942aef2b09\": container with ID starting with 887b1c1799d538caba2c3e7fe42b7234a959cb83671d40f8a5b5ea942aef2b09 not found: ID does not exist"
Nov 03 20:39:41.483265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.483234    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-podinfo\") pod \"6c85f57d-183f-408c-a450-6d6ad119a7db\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") "
Nov 03 20:39:41.483406 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.483301    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-kube-api-access-7r5ff\" (UniqueName: \"kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-kube-api-access-7r5ff\") pod \"6c85f57d-183f-408c-a450-6d6ad119a7db\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") "
Nov 03 20:39:41.483406 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.483339    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-credential-socket\") pod \"6c85f57d-183f-408c-a450-6d6ad119a7db\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") "
Nov 03 20:39:41.483406 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.483375    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-token\" (UniqueName: \"kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-token\") pod \"6c85f57d-183f-408c-a450-6d6ad119a7db\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") "
Nov 03 20:39:41.483640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.483432    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-data\") pod \"6c85f57d-183f-408c-a450-6d6ad119a7db\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") "
Nov 03 20:39:41.483640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.483597    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-socket\") pod \"6c85f57d-183f-408c-a450-6d6ad119a7db\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") "
Nov 03 20:39:41.483755 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.483683    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-credential-socket" pod "6c85f57d-183f-408c-a450-6d6ad119a7db" (UID: "6c85f57d-183f-408c-a450-6d6ad119a7db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:41.483812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.483761    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c85f57d-183f-408c-a450-6d6ad119a7db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c85f57d-183f-408c-a450-6d6ad119a7db-istiod-ca-cert\") pod \"6c85f57d-183f-408c-a450-6d6ad119a7db\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") "
Nov 03 20:39:41.483812 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.483798    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-envoy\") pod \"6c85f57d-183f-408c-a450-6d6ad119a7db\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") "
Nov 03 20:39:41.483928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.483838    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-certs\") pod \"6c85f57d-183f-408c-a450-6d6ad119a7db\" (UID: \"6c85f57d-183f-408c-a450-6d6ad119a7db\") "
Nov 03 20:39:41.483988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.483947    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-data" pod "6c85f57d-183f-408c-a450-6d6ad119a7db" (UID: "6c85f57d-183f-408c-a450-6d6ad119a7db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:41.484049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.483989    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-socket" pod "6c85f57d-183f-408c-a450-6d6ad119a7db" (UID: "6c85f57d-183f-408c-a450-6d6ad119a7db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:41.484185 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.484164    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-certs" pod "6c85f57d-183f-408c-a450-6d6ad119a7db" (UID: "6c85f57d-183f-408c-a450-6d6ad119a7db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:41.484265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.484248    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c85f57d-183f-408c-a450-6d6ad119a7db-istiod-ca-cert" pod "6c85f57d-183f-408c-a450-6d6ad119a7db" (UID: "6c85f57d-183f-408c-a450-6d6ad119a7db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:41.484477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.484455    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:41.484609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.484495    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c85f57d-183f-408c-a450-6d6ad119a7db-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:41.484609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.484511    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:41.484609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.484524    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:41.484609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.484538    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:41.486418 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.486383    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-podinfo" pod "6c85f57d-183f-408c-a450-6d6ad119a7db" (UID: "6c85f57d-183f-408c-a450-6d6ad119a7db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:39:41.486598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.486567    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-kube-api-access-7r5ff" pod "6c85f57d-183f-408c-a450-6d6ad119a7db" (UID: "6c85f57d-183f-408c-a450-6d6ad119a7db"). InnerVolumeSpecName "kube-api-access-7r5ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:41.486716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.486624    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-envoy" pod "6c85f57d-183f-408c-a450-6d6ad119a7db" (UID: "6c85f57d-183f-408c-a450-6d6ad119a7db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:41.486716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.486685    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-token" pod "6c85f57d-183f-408c-a450-6d6ad119a7db" (UID: "6c85f57d-183f-408c-a450-6d6ad119a7db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:41.511248 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.509052    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"]
Nov 03 20:39:41.583096 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.556102    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"]
Nov 03 20:39:41.595505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.593090    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:41.595505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.593128    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:41.595505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.593145    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c85f57d-183f-408c-a450-6d6ad119a7db-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:41.595505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.593160    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r5ff\" (UniqueName: \"kubernetes.io/projected/6c85f57d-183f-408c-a450-6d6ad119a7db-kube-api-access-7r5ff\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:41.628430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.628395    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:41.773887 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.770831    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:41.799816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.799464    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"]
Nov 03 20:39:41.804044 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.804008    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-cvxvh"]
Nov 03 20:39:41.971922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:41.971889    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c85f57d-183f-408c-a450-6d6ad119a7db" path="/var/lib/kubelet/pods/6c85f57d-183f-408c-a450-6d6ad119a7db/volumes"
Nov 03 20:39:42.198688 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.198601    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:42.391942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.391905    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"]
Nov 03 20:39:42.392531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.392153    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc" podUID="bd928a60-b898-49f3-8ff0-5dfc52ef4fe4" containerName="maas-api" containerID="cri-o://4a5a3154694c8844038d3f8dab150660417870576819520f01d423aeb99731d2" gracePeriod=30
Nov 03 20:39:42.549864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.549828    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:42.760920 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.760688    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:39:42.923250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.923171    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-maas-api-tls\") pod \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") "
Nov 03 20:39:42.923421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.923265    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-metrics-certs\") pod \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") "
Nov 03 20:39:42.923421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.923303    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-kube-api-access-9hcrp\" (UniqueName: \"kubernetes.io/projected/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-kube-api-access-9hcrp\") pod \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") "
Nov 03 20:39:42.923421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.923379    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-openshift-service-ca\") pod \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\" (UID: \"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4\") "
Nov 03 20:39:42.923849 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.923820    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-openshift-service-ca" pod "bd928a60-b898-49f3-8ff0-5dfc52ef4fe4" (UID: "bd928a60-b898-49f3-8ff0-5dfc52ef4fe4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:42.925541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.925515    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-metrics-certs" pod "bd928a60-b898-49f3-8ff0-5dfc52ef4fe4" (UID: "bd928a60-b898-49f3-8ff0-5dfc52ef4fe4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:42.925815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.925792    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-maas-api-tls" pod "bd928a60-b898-49f3-8ff0-5dfc52ef4fe4" (UID: "bd928a60-b898-49f3-8ff0-5dfc52ef4fe4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:42.925914 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.925887    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-kube-api-access-9hcrp" pod "bd928a60-b898-49f3-8ff0-5dfc52ef4fe4" (UID: "bd928a60-b898-49f3-8ff0-5dfc52ef4fe4"). InnerVolumeSpecName "kube-api-access-9hcrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:42.960782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:42.960744    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:43.024824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.024788    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:43.024824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.024824    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hcrp\" (UniqueName: \"kubernetes.io/projected/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-kube-api-access-9hcrp\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:43.025035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.024836    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:43.025035 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.024846    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:43.466881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.466845    5659 generic.go:356] "Generic (PLEG): container finished" podID="bd928a60-b898-49f3-8ff0-5dfc52ef4fe4" containerID="4a5a3154694c8844038d3f8dab150660417870576819520f01d423aeb99731d2" exitCode=0
Nov 03 20:39:43.467395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.466895    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"
Nov 03 20:39:43.467395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.466931    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc" event={"ID":"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4","Type":"ContainerDied","Data":"4a5a3154694c8844038d3f8dab150660417870576819520f01d423aeb99731d2"}
Nov 03 20:39:43.467395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.467001    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc" event={"ID":"bd928a60-b898-49f3-8ff0-5dfc52ef4fe4","Type":"ContainerDied","Data":"6de54aa10b8e6f7a37afb9fe46348916c82126d4b6a2278fb2b3f94c81948924"}
Nov 03 20:39:43.467395 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.467034    5659 scope.go:122] "RemoveContainer" containerID="4a5a3154694c8844038d3f8dab150660417870576819520f01d423aeb99731d2"
Nov 03 20:39:43.478344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.478327    5659 scope.go:122] "RemoveContainer" containerID="4a5a3154694c8844038d3f8dab150660417870576819520f01d423aeb99731d2"
Nov 03 20:39:43.478608 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:43.478591    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5a3154694c8844038d3f8dab150660417870576819520f01d423aeb99731d2\": container with ID starting with 4a5a3154694c8844038d3f8dab150660417870576819520f01d423aeb99731d2 not found: ID does not exist" containerID="4a5a3154694c8844038d3f8dab150660417870576819520f01d423aeb99731d2"
Nov 03 20:39:43.478666 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.478614    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5a3154694c8844038d3f8dab150660417870576819520f01d423aeb99731d2"} err="failed to get container status \"4a5a3154694c8844038d3f8dab150660417870576819520f01d423aeb99731d2\": rpc error: code = NotFound desc = could not find container \"4a5a3154694c8844038d3f8dab150660417870576819520f01d423aeb99731d2\": container with ID starting with 4a5a3154694c8844038d3f8dab150660417870576819520f01d423aeb99731d2 not found: ID does not exist"
Nov 03 20:39:43.492121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.492096    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"]
Nov 03 20:39:43.496831 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.496804    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0e4ce7-7fcdcd786d-mkspc"]
Nov 03 20:39:43.644568 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.644534    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:43.970221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:43.970184    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd928a60-b898-49f3-8ff0-5dfc52ef4fe4" path="/var/lib/kubelet/pods/bd928a60-b898-49f3-8ff0-5dfc52ef4fe4/volumes"
Nov 03 20:39:44.013110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.013080    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:44.147772 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.147726    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:44.295817 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.295726    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq" podUID="b039d7a8-14c3-4195-9967-121992120988" containerName="payload-processing" containerID="cri-o://54f85a05be55d25d64456cc4eac02566b1220c8a13a63d3b9595ed301f382bff" gracePeriod=25
Nov 03 20:39:44.296104 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.296073    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng" podUID="c397c76f-9780-496f-bb87-372a06c94d40" containerName="payload-pre-processing" containerID="cri-o://c2a7c76a41b1dbb4ec664637ef28c68f36b2e5e7f1f16102d83ab00b6c7a606e" gracePeriod=25
Nov 03 20:39:44.474203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.473360    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:44.497509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.496780    5659 generic.go:356] "Generic (PLEG): container finished" podID="b039d7a8-14c3-4195-9967-121992120988" containerID="54f85a05be55d25d64456cc4eac02566b1220c8a13a63d3b9595ed301f382bff" exitCode=0
Nov 03 20:39:44.497509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.496846    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq" event={"ID":"b039d7a8-14c3-4195-9967-121992120988","Type":"ContainerDied","Data":"54f85a05be55d25d64456cc4eac02566b1220c8a13a63d3b9595ed301f382bff"}
Nov 03 20:39:44.502183 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.498718    5659 generic.go:356] "Generic (PLEG): container finished" podID="c397c76f-9780-496f-bb87-372a06c94d40" containerID="c2a7c76a41b1dbb4ec664637ef28c68f36b2e5e7f1f16102d83ab00b6c7a606e" exitCode=0
Nov 03 20:39:44.502183 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.498836    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng" event={"ID":"c397c76f-9780-496f-bb87-372a06c94d40","Type":"ContainerDied","Data":"c2a7c76a41b1dbb4ec664637ef28c68f36b2e5e7f1f16102d83ab00b6c7a606e"}
Nov 03 20:39:44.564567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.560942    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc" podUID="376c36bb-8e35-46d3-9ced-b2a66f7370f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:39:44.624432 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.624392    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:44.663451 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.663418    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"
Nov 03 20:39:44.690717 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.690140    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"
Nov 03 20:39:44.760476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.760437    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b039d7a8-14c3-4195-9967-121992120988-kube-api-access-lk5wk\" (UniqueName: \"kubernetes.io/projected/b039d7a8-14c3-4195-9967-121992120988-kube-api-access-lk5wk\") pod \"b039d7a8-14c3-4195-9967-121992120988\" (UID: \"b039d7a8-14c3-4195-9967-121992120988\") "
Nov 03 20:39:44.760683 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.760551    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b039d7a8-14c3-4195-9967-121992120988-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b039d7a8-14c3-4195-9967-121992120988-plugins-config-volume\") pod \"b039d7a8-14c3-4195-9967-121992120988\" (UID: \"b039d7a8-14c3-4195-9967-121992120988\") "
Nov 03 20:39:44.760995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.760967    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b039d7a8-14c3-4195-9967-121992120988-plugins-config-volume" pod "b039d7a8-14c3-4195-9967-121992120988" (UID: "b039d7a8-14c3-4195-9967-121992120988"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:44.761230 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.761211    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b039d7a8-14c3-4195-9967-121992120988-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:44.762783 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.762760    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b039d7a8-14c3-4195-9967-121992120988-kube-api-access-lk5wk" pod "b039d7a8-14c3-4195-9967-121992120988" (UID: "b039d7a8-14c3-4195-9967-121992120988"). InnerVolumeSpecName "kube-api-access-lk5wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:44.860179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.860080    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"]
Nov 03 20:39:44.860409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.860337    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn" podUID="d4922de3-227d-479c-a91d-1db7e320b89b" containerName="istio-proxy" containerID="cri-o://eb646c6e8dfff224473fd85b5bffe8dde9be63216bef7649791dad8001c7783b" gracePeriod=30
Nov 03 20:39:44.862232 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.862207    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c397c76f-9780-496f-bb87-372a06c94d40-kube-api-access-xn8tw\" (UniqueName: \"kubernetes.io/projected/c397c76f-9780-496f-bb87-372a06c94d40-kube-api-access-xn8tw\") pod \"c397c76f-9780-496f-bb87-372a06c94d40\" (UID: \"c397c76f-9780-496f-bb87-372a06c94d40\") "
Nov 03 20:39:44.862425 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.862400    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c397c76f-9780-496f-bb87-372a06c94d40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c397c76f-9780-496f-bb87-372a06c94d40-plugins-config-volume\") pod \"c397c76f-9780-496f-bb87-372a06c94d40\" (UID: \"c397c76f-9780-496f-bb87-372a06c94d40\") "
Nov 03 20:39:44.862941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.862918    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk5wk\" (UniqueName: \"kubernetes.io/projected/b039d7a8-14c3-4195-9967-121992120988-kube-api-access-lk5wk\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:44.863681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.863650    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c397c76f-9780-496f-bb87-372a06c94d40-plugins-config-volume" pod "c397c76f-9780-496f-bb87-372a06c94d40" (UID: "c397c76f-9780-496f-bb87-372a06c94d40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:44.869992 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.869963    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c397c76f-9780-496f-bb87-372a06c94d40-kube-api-access-xn8tw" pod "c397c76f-9780-496f-bb87-372a06c94d40" (UID: "c397c76f-9780-496f-bb87-372a06c94d40"). InnerVolumeSpecName "kube-api-access-xn8tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:44.913258 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.913221    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:44.981505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.980691    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn8tw\" (UniqueName: \"kubernetes.io/projected/c397c76f-9780-496f-bb87-372a06c94d40-kube-api-access-xn8tw\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:44.981505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:44.980734    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c397c76f-9780-496f-bb87-372a06c94d40-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:45.217910 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:45.217874    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:39:45.442618 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.442576    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:45.508656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.508575    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"
Nov 03 20:39:45.509081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.508581    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq" event={"ID":"b039d7a8-14c3-4195-9967-121992120988","Type":"ContainerDied","Data":"137c90648dc4d2b28a9d389c7921e2a78ec2dbf1643312a90f4af312b4ec15d5"}
Nov 03 20:39:45.509081 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.508725    5659 scope.go:122] "RemoveContainer" containerID="54f85a05be55d25d64456cc4eac02566b1220c8a13a63d3b9595ed301f382bff"
Nov 03 20:39:45.510576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.510540    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng" event={"ID":"c397c76f-9780-496f-bb87-372a06c94d40","Type":"ContainerDied","Data":"6304dd6feecf6bd923f4aed74075431e71df5cc6956991faea28296229d8d15e"}
Nov 03 20:39:45.510712 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.510619    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"
Nov 03 20:39:45.521430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.521409    5659 scope.go:122] "RemoveContainer" containerID="c2a7c76a41b1dbb4ec664637ef28c68f36b2e5e7f1f16102d83ab00b6c7a606e"
Nov 03 20:39:45.540716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.540676    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"]
Nov 03 20:39:45.544548 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.544521    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e4ce7-698bd89b46-b4vbq"]
Nov 03 20:39:45.557622 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.557588    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"]
Nov 03 20:39:45.567162 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.567130    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng"]
Nov 03 20:39:45.600944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.600909    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx" podUID="fbf0a278-c34b-484d-9b5e-11243d3c2c63" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:39:45.647947 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.647904    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e4ce7-79485df6f9-mz6ng" podUID="c397c76f-9780-496f-bb87-372a06c94d40" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.133:50052\" within 1s: context deadline exceeded"
Nov 03 20:39:45.648130 ip-10-0-133-249 kubenswrapper[5659]: W1103 20:39:45.648001    5659 logging.go:55] [core] [Channel #9014 SubChannel #9015]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.133:50052", ServerName: "10.133.0.133:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.133:50052: operation was canceled"
Nov 03 20:39:45.660942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.660912    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:45.969697 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.969663    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b039d7a8-14c3-4195-9967-121992120988" path="/var/lib/kubelet/pods/b039d7a8-14c3-4195-9967-121992120988/volumes"
Nov 03 20:39:45.970177 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:45.970157    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c397c76f-9780-496f-bb87-372a06c94d40" path="/var/lib/kubelet/pods/c397c76f-9780-496f-bb87-372a06c94d40/volumes"
Nov 03 20:39:46.245670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:46.245578    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:46.385734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:46.385703    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:46.517145 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:46.517053    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:46.829749 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:46.829674    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:47.083183 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:47.083106    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:47.243506 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:47.239158    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:47.789414 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:47.789374    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:47.916959 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:47.916925    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:48.052158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:48.052066    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:48.416436 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:48.408324    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:49.131670 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:49.131636    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:49.940440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:49.940403    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:50.077102 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.077066    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:50.249859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.249825    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.281974    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-credential-socket\") pod \"d4922de3-227d-479c-a91d-1db7e320b89b\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") "
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.282054    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-data\") pod \"d4922de3-227d-479c-a91d-1db7e320b89b\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") "
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.282128    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-kube-api-access-w5kkg\" (UniqueName: \"kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-kube-api-access-w5kkg\") pod \"d4922de3-227d-479c-a91d-1db7e320b89b\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") "
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.282165    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-envoy\") pod \"d4922de3-227d-479c-a91d-1db7e320b89b\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") "
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.282222    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-istio-token\" (UniqueName: \"kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-istio-token\") pod \"d4922de3-227d-479c-a91d-1db7e320b89b\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") "
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.282257    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-certs\") pod \"d4922de3-227d-479c-a91d-1db7e320b89b\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") "
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.282375    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4922de3-227d-479c-a91d-1db7e320b89b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4922de3-227d-479c-a91d-1db7e320b89b-istio-podinfo\") pod \"d4922de3-227d-479c-a91d-1db7e320b89b\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") "
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.283134    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-credential-socket" pod "d4922de3-227d-479c-a91d-1db7e320b89b" (UID: "d4922de3-227d-479c-a91d-1db7e320b89b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.283359    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-data" pod "d4922de3-227d-479c-a91d-1db7e320b89b" (UID: "d4922de3-227d-479c-a91d-1db7e320b89b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.283594    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-socket\") pod \"d4922de3-227d-479c-a91d-1db7e320b89b\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") "
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.283688    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4922de3-227d-479c-a91d-1db7e320b89b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4922de3-227d-479c-a91d-1db7e320b89b-istiod-ca-cert\") pod \"d4922de3-227d-479c-a91d-1db7e320b89b\" (UID: \"d4922de3-227d-479c-a91d-1db7e320b89b\") "
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.284264    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.284288    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:50.284746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.284657    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4922de3-227d-479c-a91d-1db7e320b89b-istiod-ca-cert" pod "d4922de3-227d-479c-a91d-1db7e320b89b" (UID: "d4922de3-227d-479c-a91d-1db7e320b89b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:50.286047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.284924    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-socket" pod "d4922de3-227d-479c-a91d-1db7e320b89b" (UID: "d4922de3-227d-479c-a91d-1db7e320b89b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:50.286047 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.285118    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-certs" pod "d4922de3-227d-479c-a91d-1db7e320b89b" (UID: "d4922de3-227d-479c-a91d-1db7e320b89b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:50.287987 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.287957    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-istio-token" pod "d4922de3-227d-479c-a91d-1db7e320b89b" (UID: "d4922de3-227d-479c-a91d-1db7e320b89b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:50.288388 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.288352    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4922de3-227d-479c-a91d-1db7e320b89b-istio-podinfo" pod "d4922de3-227d-479c-a91d-1db7e320b89b" (UID: "d4922de3-227d-479c-a91d-1db7e320b89b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:39:50.289657 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.289626    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-kube-api-access-w5kkg" pod "d4922de3-227d-479c-a91d-1db7e320b89b" (UID: "d4922de3-227d-479c-a91d-1db7e320b89b"). InnerVolumeSpecName "kube-api-access-w5kkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:50.291976 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.291502    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-envoy" pod "d4922de3-227d-479c-a91d-1db7e320b89b" (UID: "d4922de3-227d-479c-a91d-1db7e320b89b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:50.386098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.386056    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4922de3-227d-479c-a91d-1db7e320b89b-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:50.386098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.386087    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5kkg\" (UniqueName: \"kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-kube-api-access-w5kkg\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:50.386098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.386098    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:50.386098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.386108    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4922de3-227d-479c-a91d-1db7e320b89b-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:50.386531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.386118    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:50.386531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.386126    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4922de3-227d-479c-a91d-1db7e320b89b-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:50.386531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.386134    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4922de3-227d-479c-a91d-1db7e320b89b-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:50.555922 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.555889    5659 generic.go:356] "Generic (PLEG): container finished" podID="d4922de3-227d-479c-a91d-1db7e320b89b" containerID="eb646c6e8dfff224473fd85b5bffe8dde9be63216bef7649791dad8001c7783b" exitCode=0
Nov 03 20:39:50.556110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.555937    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"
Nov 03 20:39:50.556110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.555953    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn" event={"ID":"d4922de3-227d-479c-a91d-1db7e320b89b","Type":"ContainerDied","Data":"eb646c6e8dfff224473fd85b5bffe8dde9be63216bef7649791dad8001c7783b"}
Nov 03 20:39:50.556110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.556023    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn" event={"ID":"d4922de3-227d-479c-a91d-1db7e320b89b","Type":"ContainerDied","Data":"8a81e389b1e70c8bcc4684f5c0bb9f30f88370c1339b07c7e610f563170bea00"}
Nov 03 20:39:50.556110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.556050    5659 scope.go:122] "RemoveContainer" containerID="eb646c6e8dfff224473fd85b5bffe8dde9be63216bef7649791dad8001c7783b"
Nov 03 20:39:50.568549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.568528    5659 scope.go:122] "RemoveContainer" containerID="eb646c6e8dfff224473fd85b5bffe8dde9be63216bef7649791dad8001c7783b"
Nov 03 20:39:50.568840 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:50.568816    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb646c6e8dfff224473fd85b5bffe8dde9be63216bef7649791dad8001c7783b\": container with ID starting with eb646c6e8dfff224473fd85b5bffe8dde9be63216bef7649791dad8001c7783b not found: ID does not exist" containerID="eb646c6e8dfff224473fd85b5bffe8dde9be63216bef7649791dad8001c7783b"
Nov 03 20:39:50.568933 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.568847    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb646c6e8dfff224473fd85b5bffe8dde9be63216bef7649791dad8001c7783b"} err="failed to get container status \"eb646c6e8dfff224473fd85b5bffe8dde9be63216bef7649791dad8001c7783b\": rpc error: code = NotFound desc = could not find container \"eb646c6e8dfff224473fd85b5bffe8dde9be63216bef7649791dad8001c7783b\": container with ID starting with eb646c6e8dfff224473fd85b5bffe8dde9be63216bef7649791dad8001c7783b not found: ID does not exist"
Nov 03 20:39:50.583215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.583175    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"]
Nov 03 20:39:50.585906 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.585881    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-2p4pn"]
Nov 03 20:39:50.729043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.728955    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:50.818032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.817994    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"]
Nov 03 20:39:50.939138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:50.938988    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:51.076870 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:51.076759    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:51.692689 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:51.692659    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:51.970174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:51.970137    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4922de3-227d-479c-a91d-1db7e320b89b" path="/var/lib/kubelet/pods/d4922de3-227d-479c-a91d-1db7e320b89b/volumes"
Nov 03 20:39:52.336095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:52.335972    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:52.940715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:52.940653    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:53.155132 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.155097    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"
Nov 03 20:39:53.184499 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.184436    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:53.234765 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.227650    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376c36bb-8e35-46d3-9ced-b2a66f7370f7-kube-api-access-fl9zs\" (UniqueName: \"kubernetes.io/projected/376c36bb-8e35-46d3-9ced-b2a66f7370f7-kube-api-access-fl9zs\") pod \"376c36bb-8e35-46d3-9ced-b2a66f7370f7\" (UID: \"376c36bb-8e35-46d3-9ced-b2a66f7370f7\") "
Nov 03 20:39:53.234765 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.227829    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/376c36bb-8e35-46d3-9ced-b2a66f7370f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376c36bb-8e35-46d3-9ced-b2a66f7370f7-plugins-config-volume\") pod \"376c36bb-8e35-46d3-9ced-b2a66f7370f7\" (UID: \"376c36bb-8e35-46d3-9ced-b2a66f7370f7\") "
Nov 03 20:39:53.242505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.239971    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376c36bb-8e35-46d3-9ced-b2a66f7370f7-plugins-config-volume" pod "376c36bb-8e35-46d3-9ced-b2a66f7370f7" (UID: "376c36bb-8e35-46d3-9ced-b2a66f7370f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:53.242505 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.240177    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376c36bb-8e35-46d3-9ced-b2a66f7370f7-kube-api-access-fl9zs" pod "376c36bb-8e35-46d3-9ced-b2a66f7370f7" (UID: "376c36bb-8e35-46d3-9ced-b2a66f7370f7"). InnerVolumeSpecName "kube-api-access-fl9zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:53.250822 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.250794    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"
Nov 03 20:39:53.329097 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.329063    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf0a278-c34b-484d-9b5e-11243d3c2c63-kube-api-access-qwghp\" (UniqueName: \"kubernetes.io/projected/fbf0a278-c34b-484d-9b5e-11243d3c2c63-kube-api-access-qwghp\") pod \"fbf0a278-c34b-484d-9b5e-11243d3c2c63\" (UID: \"fbf0a278-c34b-484d-9b5e-11243d3c2c63\") "
Nov 03 20:39:53.329306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.329170    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf0a278-c34b-484d-9b5e-11243d3c2c63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf0a278-c34b-484d-9b5e-11243d3c2c63-plugins-config-volume\") pod \"fbf0a278-c34b-484d-9b5e-11243d3c2c63\" (UID: \"fbf0a278-c34b-484d-9b5e-11243d3c2c63\") "
Nov 03 20:39:53.329448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.329434    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl9zs\" (UniqueName: \"kubernetes.io/projected/376c36bb-8e35-46d3-9ced-b2a66f7370f7-kube-api-access-fl9zs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:53.329557 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.329448    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/376c36bb-8e35-46d3-9ced-b2a66f7370f7-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:53.329653 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.329629    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf0a278-c34b-484d-9b5e-11243d3c2c63-plugins-config-volume" pod "fbf0a278-c34b-484d-9b5e-11243d3c2c63" (UID: "fbf0a278-c34b-484d-9b5e-11243d3c2c63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:53.331193 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.331173    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf0a278-c34b-484d-9b5e-11243d3c2c63-kube-api-access-qwghp" pod "fbf0a278-c34b-484d-9b5e-11243d3c2c63" (UID: "fbf0a278-c34b-484d-9b5e-11243d3c2c63"). InnerVolumeSpecName "kube-api-access-qwghp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:53.430678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.430589    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwghp\" (UniqueName: \"kubernetes.io/projected/fbf0a278-c34b-484d-9b5e-11243d3c2c63-kube-api-access-qwghp\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:53.430678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.430623    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf0a278-c34b-484d-9b5e-11243d3c2c63-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:53.583674 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.583626    5659 generic.go:356] "Generic (PLEG): container finished" podID="376c36bb-8e35-46d3-9ced-b2a66f7370f7" containerID="35887100fca9ea6d3ba2319a22b918fab53f2b0f873cb23f2f12cbc4027e3d4f" exitCode=0
Nov 03 20:39:53.583881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.583679    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"
Nov 03 20:39:53.583881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.583700    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc" event={"ID":"376c36bb-8e35-46d3-9ced-b2a66f7370f7","Type":"ContainerDied","Data":"35887100fca9ea6d3ba2319a22b918fab53f2b0f873cb23f2f12cbc4027e3d4f"}
Nov 03 20:39:53.583881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.583769    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc" event={"ID":"376c36bb-8e35-46d3-9ced-b2a66f7370f7","Type":"ContainerDied","Data":"136ce1abe3627ce65b8feb438eb4f6f5351ea4f8af69fad3a6491789af294420"}
Nov 03 20:39:53.583881 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.583799    5659 scope.go:122] "RemoveContainer" containerID="35887100fca9ea6d3ba2319a22b918fab53f2b0f873cb23f2f12cbc4027e3d4f"
Nov 03 20:39:53.585177 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.585150    5659 generic.go:356] "Generic (PLEG): container finished" podID="fbf0a278-c34b-484d-9b5e-11243d3c2c63" containerID="c239731ac9c2e45cd9c00b0f6862a4129ed9c55588a868ccf1d5eab48f97e263" exitCode=0
Nov 03 20:39:53.585309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.585190    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx" event={"ID":"fbf0a278-c34b-484d-9b5e-11243d3c2c63","Type":"ContainerDied","Data":"c239731ac9c2e45cd9c00b0f6862a4129ed9c55588a868ccf1d5eab48f97e263"}
Nov 03 20:39:53.585309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.585252    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx" event={"ID":"fbf0a278-c34b-484d-9b5e-11243d3c2c63","Type":"ContainerDied","Data":"193bcd2f715278e40bcf4db7606ce6dd6c52b89900f207af3b1aedfa02ad0406"}
Nov 03 20:39:53.585309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.585203    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"
Nov 03 20:39:53.602124 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.602098    5659 scope.go:122] "RemoveContainer" containerID="35887100fca9ea6d3ba2319a22b918fab53f2b0f873cb23f2f12cbc4027e3d4f"
Nov 03 20:39:53.602760 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:53.602735    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35887100fca9ea6d3ba2319a22b918fab53f2b0f873cb23f2f12cbc4027e3d4f\": container with ID starting with 35887100fca9ea6d3ba2319a22b918fab53f2b0f873cb23f2f12cbc4027e3d4f not found: ID does not exist" containerID="35887100fca9ea6d3ba2319a22b918fab53f2b0f873cb23f2f12cbc4027e3d4f"
Nov 03 20:39:53.602865 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.602764    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35887100fca9ea6d3ba2319a22b918fab53f2b0f873cb23f2f12cbc4027e3d4f"} err="failed to get container status \"35887100fca9ea6d3ba2319a22b918fab53f2b0f873cb23f2f12cbc4027e3d4f\": rpc error: code = NotFound desc = could not find container \"35887100fca9ea6d3ba2319a22b918fab53f2b0f873cb23f2f12cbc4027e3d4f\": container with ID starting with 35887100fca9ea6d3ba2319a22b918fab53f2b0f873cb23f2f12cbc4027e3d4f not found: ID does not exist"
Nov 03 20:39:53.602865 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.602783    5659 scope.go:122] "RemoveContainer" containerID="c239731ac9c2e45cd9c00b0f6862a4129ed9c55588a868ccf1d5eab48f97e263"
Nov 03 20:39:53.613550 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.613529    5659 scope.go:122] "RemoveContainer" containerID="c239731ac9c2e45cd9c00b0f6862a4129ed9c55588a868ccf1d5eab48f97e263"
Nov 03 20:39:53.613737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.613716    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"]
Nov 03 20:39:53.613827 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:53.613810    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c239731ac9c2e45cd9c00b0f6862a4129ed9c55588a868ccf1d5eab48f97e263\": container with ID starting with c239731ac9c2e45cd9c00b0f6862a4129ed9c55588a868ccf1d5eab48f97e263 not found: ID does not exist" containerID="c239731ac9c2e45cd9c00b0f6862a4129ed9c55588a868ccf1d5eab48f97e263"
Nov 03 20:39:53.613873 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.613836    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c239731ac9c2e45cd9c00b0f6862a4129ed9c55588a868ccf1d5eab48f97e263"} err="failed to get container status \"c239731ac9c2e45cd9c00b0f6862a4129ed9c55588a868ccf1d5eab48f97e263\": rpc error: code = NotFound desc = could not find container \"c239731ac9c2e45cd9c00b0f6862a4129ed9c55588a868ccf1d5eab48f97e263\": container with ID starting with c239731ac9c2e45cd9c00b0f6862a4129ed9c55588a868ccf1d5eab48f97e263 not found: ID does not exist"
Nov 03 20:39:53.620330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.620306    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-660d95-84bf7f47d7-q7ncx"]
Nov 03 20:39:53.631883 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.631855    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"]
Nov 03 20:39:53.637471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.637447    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-660d95-8474b9648b-78rhc"]
Nov 03 20:39:53.919474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.919437    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:53.980508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.977467    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="376c36bb-8e35-46d3-9ced-b2a66f7370f7" path="/var/lib/kubelet/pods/376c36bb-8e35-46d3-9ced-b2a66f7370f7/volumes"
Nov 03 20:39:53.980508 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:53.978159    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf0a278-c34b-484d-9b5e-11243d3c2c63" path="/var/lib/kubelet/pods/fbf0a278-c34b-484d-9b5e-11243d3c2c63/volumes"
Nov 03 20:39:54.121751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:54.121716    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:54.266053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:54.266019    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:54.804504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:54.804110    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:55.290912 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:55.290870    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:55.414750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:55.414714    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:55.668758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:55.668666    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:55.819275 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:55.819230    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm" podUID="57bc1738-e920-4570-b31b-9e4937df03b0" containerName="payload-processing" containerID="cri-o://356242d5761fd5d4d8b9b58bb7d5f2fc8fc55c88f8007beaa21b9fe861afb516" gracePeriod=25
Nov 03 20:39:56.089002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.088974    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"
Nov 03 20:39:56.172944 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.172907    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57bc1738-e920-4570-b31b-9e4937df03b0-kube-api-access-4dq5l\" (UniqueName: \"kubernetes.io/projected/57bc1738-e920-4570-b31b-9e4937df03b0-kube-api-access-4dq5l\") pod \"57bc1738-e920-4570-b31b-9e4937df03b0\" (UID: \"57bc1738-e920-4570-b31b-9e4937df03b0\") "
Nov 03 20:39:56.173197 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.173176    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57bc1738-e920-4570-b31b-9e4937df03b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57bc1738-e920-4570-b31b-9e4937df03b0-plugins-config-volume\") pod \"57bc1738-e920-4570-b31b-9e4937df03b0\" (UID: \"57bc1738-e920-4570-b31b-9e4937df03b0\") "
Nov 03 20:39:56.173627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.173598    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57bc1738-e920-4570-b31b-9e4937df03b0-plugins-config-volume" pod "57bc1738-e920-4570-b31b-9e4937df03b0" (UID: "57bc1738-e920-4570-b31b-9e4937df03b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:56.175754 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.175725    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bc1738-e920-4570-b31b-9e4937df03b0-kube-api-access-4dq5l" pod "57bc1738-e920-4570-b31b-9e4937df03b0" (UID: "57bc1738-e920-4570-b31b-9e4937df03b0"). InnerVolumeSpecName "kube-api-access-4dq5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:56.247038 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.247000    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:56.274409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.274373    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dq5l\" (UniqueName: \"kubernetes.io/projected/57bc1738-e920-4570-b31b-9e4937df03b0-kube-api-access-4dq5l\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:56.274409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.274403    5659 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57bc1738-e920-4570-b31b-9e4937df03b0-plugins-config-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:56.386325 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.383003    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:56.566377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.566337    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:56.627215 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.627160    5659 generic.go:356] "Generic (PLEG): container finished" podID="57bc1738-e920-4570-b31b-9e4937df03b0" containerID="356242d5761fd5d4d8b9b58bb7d5f2fc8fc55c88f8007beaa21b9fe861afb516" exitCode=0
Nov 03 20:39:56.627441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.627214    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"
Nov 03 20:39:56.627441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.627298    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm" event={"ID":"57bc1738-e920-4570-b31b-9e4937df03b0","Type":"ContainerDied","Data":"356242d5761fd5d4d8b9b58bb7d5f2fc8fc55c88f8007beaa21b9fe861afb516"}
Nov 03 20:39:56.627441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.627357    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm" event={"ID":"57bc1738-e920-4570-b31b-9e4937df03b0","Type":"ContainerDied","Data":"c620612e0d2249a30fea0d358b747ef55927aa5b02f49ba4c0a3e0b9d647aef7"}
Nov 03 20:39:56.627441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.627390    5659 scope.go:122] "RemoveContainer" containerID="356242d5761fd5d4d8b9b58bb7d5f2fc8fc55c88f8007beaa21b9fe861afb516"
Nov 03 20:39:56.639709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.639688    5659 scope.go:122] "RemoveContainer" containerID="356242d5761fd5d4d8b9b58bb7d5f2fc8fc55c88f8007beaa21b9fe861afb516"
Nov 03 20:39:56.640020 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:56.639998    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"356242d5761fd5d4d8b9b58bb7d5f2fc8fc55c88f8007beaa21b9fe861afb516\": container with ID starting with 356242d5761fd5d4d8b9b58bb7d5f2fc8fc55c88f8007beaa21b9fe861afb516 not found: ID does not exist" containerID="356242d5761fd5d4d8b9b58bb7d5f2fc8fc55c88f8007beaa21b9fe861afb516"
Nov 03 20:39:56.640106 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.640034    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356242d5761fd5d4d8b9b58bb7d5f2fc8fc55c88f8007beaa21b9fe861afb516"} err="failed to get container status \"356242d5761fd5d4d8b9b58bb7d5f2fc8fc55c88f8007beaa21b9fe861afb516\": rpc error: code = NotFound desc = could not find container \"356242d5761fd5d4d8b9b58bb7d5f2fc8fc55c88f8007beaa21b9fe861afb516\": container with ID starting with 356242d5761fd5d4d8b9b58bb7d5f2fc8fc55c88f8007beaa21b9fe861afb516 not found: ID does not exist"
Nov 03 20:39:56.647541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.647511    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main" containerID="cri-o://17ae96428fafb8554b2277c319828bfed909fcc7d2d6239aa6a6cee9dd3c51e4" gracePeriod=45
Nov 03 20:39:56.653982 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.653957    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"]
Nov 03 20:39:56.659456 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.659431    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-43c674-bf97746fd-vzmjm"]
Nov 03 20:39:56.666796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.666756    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" containerID="cri-o://79a6a31d347d053fdcc2425aa4f98e70f5def05756a62dfd189a6bcbb02a1103" gracePeriod=45
Nov 03 20:39:56.676547 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.676521    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": dial tcp 10.133.0.63:8000: connect: connection refused"
Nov 03 20:39:56.944828 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:56.944801    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:39:57.039866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.039828    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 20:39:57.108897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.108818    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72080325-72f5-4713-a185-570a01746fc8-kube-api-access-4zhpl\" (UniqueName: \"kubernetes.io/projected/72080325-72f5-4713-a185-570a01746fc8-kube-api-access-4zhpl\") pod \"72080325-72f5-4713-a185-570a01746fc8\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") "
Nov 03 20:39:57.109080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.108905    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-dshm\") pod \"72080325-72f5-4713-a185-570a01746fc8\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") "
Nov 03 20:39:57.109080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.109024    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-home\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-home\") pod \"72080325-72f5-4713-a185-570a01746fc8\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") "
Nov 03 20:39:57.109212 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.109085    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72080325-72f5-4713-a185-570a01746fc8-tls-certs\" (UniqueName: \"kubernetes.io/secret/72080325-72f5-4713-a185-570a01746fc8-tls-certs\") pod \"72080325-72f5-4713-a185-570a01746fc8\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") "
Nov 03 20:39:57.109281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.109263    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-tmp-dir\") pod \"72080325-72f5-4713-a185-570a01746fc8\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") "
Nov 03 20:39:57.109354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.109300    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-home" pod "72080325-72f5-4713-a185-570a01746fc8" (UID: "72080325-72f5-4713-a185-570a01746fc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:57.109354 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.109326    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-model-cache\") pod \"72080325-72f5-4713-a185-570a01746fc8\" (UID: \"72080325-72f5-4713-a185-570a01746fc8\") "
Nov 03 20:39:57.109681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.109510    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-tmp-dir" pod "72080325-72f5-4713-a185-570a01746fc8" (UID: "72080325-72f5-4713-a185-570a01746fc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:57.109681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.109647    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-model-cache" pod "72080325-72f5-4713-a185-570a01746fc8" (UID: "72080325-72f5-4713-a185-570a01746fc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:57.109979 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.109956    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.110060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.110007    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.110060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.110024    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.111137 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.111113    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-dshm" pod "72080325-72f5-4713-a185-570a01746fc8" (UID: "72080325-72f5-4713-a185-570a01746fc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:57.111334 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.111318    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72080325-72f5-4713-a185-570a01746fc8-tls-certs" pod "72080325-72f5-4713-a185-570a01746fc8" (UID: "72080325-72f5-4713-a185-570a01746fc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:57.111563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.111547    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72080325-72f5-4713-a185-570a01746fc8-kube-api-access-4zhpl" pod "72080325-72f5-4713-a185-570a01746fc8" (UID: "72080325-72f5-4713-a185-570a01746fc8"). InnerVolumeSpecName "kube-api-access-4zhpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:57.211093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.211060    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tmp-dir\") pod \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") "
Nov 03 20:39:57.211282 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.211121    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tls-certs\") pod \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") "
Nov 03 20:39:57.211282 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.211231    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-home\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-home\") pod \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") "
Nov 03 20:39:57.211412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.211300    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-dshm\") pod \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") "
Nov 03 20:39:57.211412 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.211332    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tmp-dir" pod "f2d9f7b7-51af-4be2-b959-3143ebc16f58" (UID: "f2d9f7b7-51af-4be2-b959-3143ebc16f58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:57.211668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.211417    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d9f7b7-51af-4be2-b959-3143ebc16f58-kube-api-access-nkvqw\" (UniqueName: \"kubernetes.io/projected/f2d9f7b7-51af-4be2-b959-3143ebc16f58-kube-api-access-nkvqw\") pod \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") "
Nov 03 20:39:57.211668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.211465    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-home" pod "f2d9f7b7-51af-4be2-b959-3143ebc16f58" (UID: "f2d9f7b7-51af-4be2-b959-3143ebc16f58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:57.211668 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.211476    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-model-cache\") pod \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\" (UID: \"f2d9f7b7-51af-4be2-b959-3143ebc16f58\") "
Nov 03 20:39:57.211855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.211759    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-model-cache" pod "f2d9f7b7-51af-4be2-b959-3143ebc16f58" (UID: "f2d9f7b7-51af-4be2-b959-3143ebc16f58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:57.212020 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.211994    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zhpl\" (UniqueName: \"kubernetes.io/projected/72080325-72f5-4713-a185-570a01746fc8-kube-api-access-4zhpl\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.212152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.212024    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.212152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.212038    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72080325-72f5-4713-a185-570a01746fc8-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.212152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.212047    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72080325-72f5-4713-a185-570a01746fc8-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.212152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.212055    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.212152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.212063    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.213534 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.213510    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tls-certs" pod "f2d9f7b7-51af-4be2-b959-3143ebc16f58" (UID: "f2d9f7b7-51af-4be2-b959-3143ebc16f58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:57.213620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.213542    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-dshm" pod "f2d9f7b7-51af-4be2-b959-3143ebc16f58" (UID: "f2d9f7b7-51af-4be2-b959-3143ebc16f58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:57.213620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.213557    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d9f7b7-51af-4be2-b959-3143ebc16f58-kube-api-access-nkvqw" pod "f2d9f7b7-51af-4be2-b959-3143ebc16f58" (UID: "f2d9f7b7-51af-4be2-b959-3143ebc16f58"). InnerVolumeSpecName "kube-api-access-nkvqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:57.313281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.313241    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkvqw\" (UniqueName: \"kubernetes.io/projected/f2d9f7b7-51af-4be2-b959-3143ebc16f58-kube-api-access-nkvqw\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.313281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.313276    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d9f7b7-51af-4be2-b959-3143ebc16f58-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.313281 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.313286    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d9f7b7-51af-4be2-b959-3143ebc16f58-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.323750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.323720    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:57.636518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.636224    5659 generic.go:356] "Generic (PLEG): container finished" podID="72080325-72f5-4713-a185-570a01746fc8" containerID="17ae96428fafb8554b2277c319828bfed909fcc7d2d6239aa6a6cee9dd3c51e4" exitCode=0
Nov 03 20:39:57.636518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.636261    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" event={"ID":"72080325-72f5-4713-a185-570a01746fc8","Type":"ContainerDied","Data":"17ae96428fafb8554b2277c319828bfed909fcc7d2d6239aa6a6cee9dd3c51e4"}
Nov 03 20:39:57.636518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.636282    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"
Nov 03 20:39:57.636518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.636382    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw" event={"ID":"72080325-72f5-4713-a185-570a01746fc8","Type":"ContainerDied","Data":"df5b56d9a52ba99dd96f61fe61a94d25f8c3a88efaf9fa2876e4e11eb7a32139"}
Nov 03 20:39:57.636518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.636392    5659 scope.go:122] "RemoveContainer" containerID="17ae96428fafb8554b2277c319828bfed909fcc7d2d6239aa6a6cee9dd3c51e4"
Nov 03 20:39:57.639191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.639165    5659 generic.go:356] "Generic (PLEG): container finished" podID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerID="79a6a31d347d053fdcc2425aa4f98e70f5def05756a62dfd189a6bcbb02a1103" exitCode=0
Nov 03 20:39:57.639336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.639237    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"
Nov 03 20:39:57.639336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.639246    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" event={"ID":"f2d9f7b7-51af-4be2-b959-3143ebc16f58","Type":"ContainerDied","Data":"79a6a31d347d053fdcc2425aa4f98e70f5def05756a62dfd189a6bcbb02a1103"}
Nov 03 20:39:57.639336 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.639298    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t" event={"ID":"f2d9f7b7-51af-4be2-b959-3143ebc16f58","Type":"ContainerDied","Data":"edc4473cced726887611d9752390f3308e25b56f029c0373b9cecbfa171e1042"}
Nov 03 20:39:57.647928 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.647911    5659 scope.go:122] "RemoveContainer" containerID="17ae96428fafb8554b2277c319828bfed909fcc7d2d6239aa6a6cee9dd3c51e4"
Nov 03 20:39:57.648198 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:57.648177    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ae96428fafb8554b2277c319828bfed909fcc7d2d6239aa6a6cee9dd3c51e4\": container with ID starting with 17ae96428fafb8554b2277c319828bfed909fcc7d2d6239aa6a6cee9dd3c51e4 not found: ID does not exist" containerID="17ae96428fafb8554b2277c319828bfed909fcc7d2d6239aa6a6cee9dd3c51e4"
Nov 03 20:39:57.648270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.648205    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ae96428fafb8554b2277c319828bfed909fcc7d2d6239aa6a6cee9dd3c51e4"} err="failed to get container status \"17ae96428fafb8554b2277c319828bfed909fcc7d2d6239aa6a6cee9dd3c51e4\": rpc error: code = NotFound desc = could not find container \"17ae96428fafb8554b2277c319828bfed909fcc7d2d6239aa6a6cee9dd3c51e4\": container with ID starting with 17ae96428fafb8554b2277c319828bfed909fcc7d2d6239aa6a6cee9dd3c51e4 not found: ID does not exist"
Nov 03 20:39:57.648270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.648230    5659 scope.go:122] "RemoveContainer" containerID="79a6a31d347d053fdcc2425aa4f98e70f5def05756a62dfd189a6bcbb02a1103"
Nov 03 20:39:57.658491 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.658450    5659 scope.go:122] "RemoveContainer" containerID="79a6a31d347d053fdcc2425aa4f98e70f5def05756a62dfd189a6bcbb02a1103"
Nov 03 20:39:57.658876 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:39:57.658847    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a6a31d347d053fdcc2425aa4f98e70f5def05756a62dfd189a6bcbb02a1103\": container with ID starting with 79a6a31d347d053fdcc2425aa4f98e70f5def05756a62dfd189a6bcbb02a1103 not found: ID does not exist" containerID="79a6a31d347d053fdcc2425aa4f98e70f5def05756a62dfd189a6bcbb02a1103"
Nov 03 20:39:57.659020 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.658884    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a6a31d347d053fdcc2425aa4f98e70f5def05756a62dfd189a6bcbb02a1103"} err="failed to get container status \"79a6a31d347d053fdcc2425aa4f98e70f5def05756a62dfd189a6bcbb02a1103\": rpc error: code = NotFound desc = could not find container \"79a6a31d347d053fdcc2425aa4f98e70f5def05756a62dfd189a6bcbb02a1103\": container with ID starting with 79a6a31d347d053fdcc2425aa4f98e70f5def05756a62dfd189a6bcbb02a1103 not found: ID does not exist"
Nov 03 20:39:57.667589 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.667564    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"]
Nov 03 20:39:57.668474 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.668452    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/simulator-w2-2b5703-kserve-55d9595866-ccrnw"]
Nov 03 20:39:57.678439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.678421    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"]
Nov 03 20:39:57.684649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.684623    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-2-w2-2b5703-kserve-56bd689fbf-nsw9t"]
Nov 03 20:39:57.971297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.971261    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57bc1738-e920-4570-b31b-9e4937df03b0" path="/var/lib/kubelet/pods/57bc1738-e920-4570-b31b-9e4937df03b0/volumes"
Nov 03 20:39:57.971836 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.971815    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72080325-72f5-4713-a185-570a01746fc8" path="/var/lib/kubelet/pods/72080325-72f5-4713-a185-570a01746fc8/volumes"
Nov 03 20:39:57.972311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:57.972291    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" path="/var/lib/kubelet/pods/f2d9f7b7-51af-4be2-b959-3143ebc16f58/volumes"
Nov 03 20:39:58.033891 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:58.033854    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:58.733754 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:58.733719    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:59.145946 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:59.145862    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:59.284026 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:59.283993    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:59.478576 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:59.478530    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:59.661820 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:59.661784    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:59.914362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:59.913340    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"]
Nov 03 20:39:59.914362 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:39:59.913611    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5" podUID="be08436f-abba-4a9d-9aa3-1f9aa05d883e" containerName="istio-proxy" containerID="cri-o://e8857a72f77126105a4fb861bc43dab83ecf9c5ae7a5c7d3e356e2986f92bb5d" gracePeriod=30
Nov 03 20:40:00.023729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:00.023679    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:00.305426 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:00.305389    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:00.432457 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:00.432362    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:00.577272 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:00.577195    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:01.360329 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:01.359720    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:02.050376 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:02.050337    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:02.646352 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:02.646306    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:02.834158 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:02.834121    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:03.031095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:03.030949    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:03.180198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:03.180159    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:03.552221 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:03.552170    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:03.748723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:03.748632    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:03.882908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:03.882811    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:04.148677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:04.148565    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:04.435274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:04.435179    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:04.994726 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:04.994688    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:05.222695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.222666    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:40:05.277885 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.277850    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-credential-socket\") pod \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") "
Nov 03 20:40:05.278071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.277926    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-certs\") pod \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") "
Nov 03 20:40:05.278071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.278060    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-envoy\") pod \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") "
Nov 03 20:40:05.278214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.278106    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-data\") pod \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") "
Nov 03 20:40:05.278214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.278152    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-podinfo\") pod \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") "
Nov 03 20:40:05.278214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.278183    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-token\" (UniqueName: \"kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-token\") pod \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") "
Nov 03 20:40:05.278379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.278252    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istiod-ca-cert\") pod \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") "
Nov 03 20:40:05.278379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.278281    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-kube-api-access-ptljm\" (UniqueName: \"kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-kube-api-access-ptljm\") pod \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") "
Nov 03 20:40:05.278379 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.278311    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-socket\") pod \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\" (UID: \"be08436f-abba-4a9d-9aa3-1f9aa05d883e\") "
Nov 03 20:40:05.279017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.278865    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-data" pod "be08436f-abba-4a9d-9aa3-1f9aa05d883e" (UID: "be08436f-abba-4a9d-9aa3-1f9aa05d883e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:05.279017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.278874    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-socket" pod "be08436f-abba-4a9d-9aa3-1f9aa05d883e" (UID: "be08436f-abba-4a9d-9aa3-1f9aa05d883e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:05.279244 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.279184    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-certs" pod "be08436f-abba-4a9d-9aa3-1f9aa05d883e" (UID: "be08436f-abba-4a9d-9aa3-1f9aa05d883e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:05.279398 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.279365    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istiod-ca-cert" pod "be08436f-abba-4a9d-9aa3-1f9aa05d883e" (UID: "be08436f-abba-4a9d-9aa3-1f9aa05d883e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:05.279877 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.279854    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-credential-socket" pod "be08436f-abba-4a9d-9aa3-1f9aa05d883e" (UID: "be08436f-abba-4a9d-9aa3-1f9aa05d883e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:05.281704 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.281682    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-podinfo" pod "be08436f-abba-4a9d-9aa3-1f9aa05d883e" (UID: "be08436f-abba-4a9d-9aa3-1f9aa05d883e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:40:05.281816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.281729    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-token" pod "be08436f-abba-4a9d-9aa3-1f9aa05d883e" (UID: "be08436f-abba-4a9d-9aa3-1f9aa05d883e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:05.281874 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.281831    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-envoy" pod "be08436f-abba-4a9d-9aa3-1f9aa05d883e" (UID: "be08436f-abba-4a9d-9aa3-1f9aa05d883e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:05.282010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.281993    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-kube-api-access-ptljm" pod "be08436f-abba-4a9d-9aa3-1f9aa05d883e" (UID: "be08436f-abba-4a9d-9aa3-1f9aa05d883e"). InnerVolumeSpecName "kube-api-access-ptljm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:05.348540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.348493    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"]
Nov 03 20:40:05.373609 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.372509    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"]
Nov 03 20:40:05.384953 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.384923    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.385192 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.385174    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.385287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.385275    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.385375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.385355    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.385375 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.385376    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be08436f-abba-4a9d-9aa3-1f9aa05d883e-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.385604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.385390    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptljm\" (UniqueName: \"kubernetes.io/projected/be08436f-abba-4a9d-9aa3-1f9aa05d883e-kube-api-access-ptljm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.385604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.385399    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.385604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.385409    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.385604 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.385418    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be08436f-abba-4a9d-9aa3-1f9aa05d883e-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.516419 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.516384    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:05.764504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.758657    5659 generic.go:356] "Generic (PLEG): container finished" podID="be08436f-abba-4a9d-9aa3-1f9aa05d883e" containerID="e8857a72f77126105a4fb861bc43dab83ecf9c5ae7a5c7d3e356e2986f92bb5d" exitCode=0
Nov 03 20:40:05.764504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.758703    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5" event={"ID":"be08436f-abba-4a9d-9aa3-1f9aa05d883e","Type":"ContainerDied","Data":"e8857a72f77126105a4fb861bc43dab83ecf9c5ae7a5c7d3e356e2986f92bb5d"}
Nov 03 20:40:05.764504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.758741    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"
Nov 03 20:40:05.764504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.758806    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5" event={"ID":"be08436f-abba-4a9d-9aa3-1f9aa05d883e","Type":"ContainerDied","Data":"0869b58d57a53d160cdc01eb058358dbf1b1cf429a873e964ed4c25442598314"}
Nov 03 20:40:05.764504 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.758837    5659 scope.go:122] "RemoveContainer" containerID="e8857a72f77126105a4fb861bc43dab83ecf9c5ae7a5c7d3e356e2986f92bb5d"
Nov 03 20:40:05.776980 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.776953    5659 scope.go:122] "RemoveContainer" containerID="e8857a72f77126105a4fb861bc43dab83ecf9c5ae7a5c7d3e356e2986f92bb5d"
Nov 03 20:40:05.777314 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:40:05.777293    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8857a72f77126105a4fb861bc43dab83ecf9c5ae7a5c7d3e356e2986f92bb5d\": container with ID starting with e8857a72f77126105a4fb861bc43dab83ecf9c5ae7a5c7d3e356e2986f92bb5d not found: ID does not exist" containerID="e8857a72f77126105a4fb861bc43dab83ecf9c5ae7a5c7d3e356e2986f92bb5d"
Nov 03 20:40:05.777407 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.777323    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8857a72f77126105a4fb861bc43dab83ecf9c5ae7a5c7d3e356e2986f92bb5d"} err="failed to get container status \"e8857a72f77126105a4fb861bc43dab83ecf9c5ae7a5c7d3e356e2986f92bb5d\": rpc error: code = NotFound desc = could not find container \"e8857a72f77126105a4fb861bc43dab83ecf9c5ae7a5c7d3e356e2986f92bb5d\": container with ID starting with e8857a72f77126105a4fb861bc43dab83ecf9c5ae7a5c7d3e356e2986f92bb5d not found: ID does not exist"
Nov 03 20:40:05.793245 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.793203    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"]
Nov 03 20:40:05.804819 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.804779    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-jhdp5"]
Nov 03 20:40:05.966213 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.956403    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:05.983838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:05.983791    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be08436f-abba-4a9d-9aa3-1f9aa05d883e" path="/var/lib/kubelet/pods/be08436f-abba-4a9d-9aa3-1f9aa05d883e/volumes"
Nov 03 20:40:06.099681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:06.099596    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:06.514239 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:06.514201    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:06.719417 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:06.719369    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:07.467737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:07.467697    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:08.278502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:08.260182    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:09.062174 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:09.062140    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:09.845746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:09.845709    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:09.969442 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:09.969409    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:10.557022 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:10.556982    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:10.944186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:10.944107    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:11.892198 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:11.891441    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:12.415792 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:12.415760    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:12.629078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:12.629036    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:12.764751 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:12.764719    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:13.460119 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:13.460082    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:13.590591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:13.590559    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:14.157514 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:14.149886    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:14.856803 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:14.856770    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:15.218211 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:40:15.218172    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:40:15.543789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:15.543694    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:16.260002 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:16.259958    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:17.267801 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:17.267764    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:18.205312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:18.205271    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"]
Nov 03 20:40:18.205634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:18.205570    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" podUID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" containerName="istio-proxy" containerID="cri-o://668fd6d24c0fb31ea276a6cfe69f0c2ee33384ca7e5d57a6b4aaab31948857db" gracePeriod=30
Nov 03 20:40:18.267858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:18.267822    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:19.052863 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:19.052813    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:19.500493 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:19.500449    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:19.638100 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:19.638067    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:19.777562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:19.777450    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:19.911876 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:19.911838    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:20.295177 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.295143    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:20.370430 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.370387    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" containerID="cri-o://9c5473d2d956c8eac1e6ed21263c2deaab3911e14804a6104c210aa0938f23c6" gracePeriod=45
Nov 03 20:40:20.402247 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.402215    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" containerID="cri-o://0ef467b5b183c95c0eb03b83b3425b091c627eb44f27af52a12462fb07499ef1" gracePeriod=45
Nov 03 20:40:20.506757 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.506719    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"]
Nov 03 20:40:20.507345 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.506976    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf" podUID="bee89175-06af-4e58-8b04-bd892b530f1e" containerName="istio-proxy" containerID="cri-o://8b95a5948ef9b60fc35d20919ab0ad9f743cab22f5c1f57f89e65c64554c74e7" gracePeriod=30
Nov 03 20:40:20.529248 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.529216    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": dial tcp 10.133.0.60:8000: connect: connection refused"
Nov 03 20:40:20.591257 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.591212    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:20.660372 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.658320    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": dial tcp 10.133.0.50:8000: connect: connection refused"
Nov 03 20:40:20.750937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.750861    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:40:20.794440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.794410    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:40:20.900032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.899954    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-home\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-home\") pod \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") "
Nov 03 20:40:20.900032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900002    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-model-cache\") pod \"92ffe4fd-3411-41b3-898f-44da3191ee57\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") "
Nov 03 20:40:20.900032 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900034    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-tmp-dir\") pod \"92ffe4fd-3411-41b3-898f-44da3191ee57\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") "
Nov 03 20:40:20.900328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900075    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tls-certs\") pod \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") "
Nov 03 20:40:20.900328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900127    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tmp-dir\") pod \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") "
Nov 03 20:40:20.900328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900223    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-kube-api-access-bzdl8\" (UniqueName: \"kubernetes.io/projected/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-kube-api-access-bzdl8\") pod \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") "
Nov 03 20:40:20.900328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900264    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-dshm\") pod \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") "
Nov 03 20:40:20.900328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900261    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-home" pod "0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" (UID: "0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.900328 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900326    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ffe4fd-3411-41b3-898f-44da3191ee57-kube-api-access-rcldh\" (UniqueName: \"kubernetes.io/projected/92ffe4fd-3411-41b3-898f-44da3191ee57-kube-api-access-rcldh\") pod \"92ffe4fd-3411-41b3-898f-44da3191ee57\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") "
Nov 03 20:40:20.900672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900341    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-model-cache" pod "92ffe4fd-3411-41b3-898f-44da3191ee57" (UID: "92ffe4fd-3411-41b3-898f-44da3191ee57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.900672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900349    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-tmp-dir" pod "92ffe4fd-3411-41b3-898f-44da3191ee57" (UID: "92ffe4fd-3411-41b3-898f-44da3191ee57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.900672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900374    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-home\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-home\") pod \"92ffe4fd-3411-41b3-898f-44da3191ee57\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") "
Nov 03 20:40:20.900672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900453    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-model-cache\") pod \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\" (UID: \"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9\") "
Nov 03 20:40:20.900672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900508    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-dshm\") pod \"92ffe4fd-3411-41b3-898f-44da3191ee57\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") "
Nov 03 20:40:20.900672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900583    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ffe4fd-3411-41b3-898f-44da3191ee57-tls-certs\" (UniqueName: \"kubernetes.io/secret/92ffe4fd-3411-41b3-898f-44da3191ee57-tls-certs\") pod \"92ffe4fd-3411-41b3-898f-44da3191ee57\" (UID: \"92ffe4fd-3411-41b3-898f-44da3191ee57\") "
Nov 03 20:40:20.900672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900653    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-home" pod "92ffe4fd-3411-41b3-898f-44da3191ee57" (UID: "92ffe4fd-3411-41b3-898f-44da3191ee57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.901076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.900826    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tmp-dir" pod "0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" (UID: "0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.901140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.901080    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.901140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.901097    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.901140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.901107    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.901140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.901116    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.901140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.901124    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.901402 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.901383    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-model-cache" pod "0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" (UID: "0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.902968 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.902943    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-dshm" pod "0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" (UID: "0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.903109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.902975    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ffe4fd-3411-41b3-898f-44da3191ee57-kube-api-access-rcldh" pod "92ffe4fd-3411-41b3-898f-44da3191ee57" (UID: "92ffe4fd-3411-41b3-898f-44da3191ee57"). InnerVolumeSpecName "kube-api-access-rcldh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:20.903109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.903032    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-kube-api-access-bzdl8" pod "0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" (UID: "0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9"). InnerVolumeSpecName "kube-api-access-bzdl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:20.903109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.903075    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ffe4fd-3411-41b3-898f-44da3191ee57-tls-certs" pod "92ffe4fd-3411-41b3-898f-44da3191ee57" (UID: "92ffe4fd-3411-41b3-898f-44da3191ee57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:20.903304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.903195    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-dshm" pod "92ffe4fd-3411-41b3-898f-44da3191ee57" (UID: "92ffe4fd-3411-41b3-898f-44da3191ee57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.903424 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.903405    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tls-certs" pod "0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" (UID: "0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:20.928477 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.928450    5659 generic.go:356] "Generic (PLEG): container finished" podID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerID="9c5473d2d956c8eac1e6ed21263c2deaab3911e14804a6104c210aa0938f23c6" exitCode=0
Nov 03 20:40:20.928635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.928512    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" event={"ID":"92ffe4fd-3411-41b3-898f-44da3191ee57","Type":"ContainerDied","Data":"9c5473d2d956c8eac1e6ed21263c2deaab3911e14804a6104c210aa0938f23c6"}
Nov 03 20:40:20.928635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.928521    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"
Nov 03 20:40:20.928635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.928554    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf" event={"ID":"92ffe4fd-3411-41b3-898f-44da3191ee57","Type":"ContainerDied","Data":"e6874fc5cd449a9c2675f5ad61524749f7178b297f3e2568da7b2266d8baf436"}
Nov 03 20:40:20.928635 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.928575    5659 scope.go:122] "RemoveContainer" containerID="9c5473d2d956c8eac1e6ed21263c2deaab3911e14804a6104c210aa0938f23c6"
Nov 03 20:40:20.930513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.930454    5659 generic.go:356] "Generic (PLEG): container finished" podID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerID="0ef467b5b183c95c0eb03b83b3425b091c627eb44f27af52a12462fb07499ef1" exitCode=0
Nov 03 20:40:20.930513 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.930510    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"
Nov 03 20:40:20.930675 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.930552    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" event={"ID":"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9","Type":"ContainerDied","Data":"0ef467b5b183c95c0eb03b83b3425b091c627eb44f27af52a12462fb07499ef1"}
Nov 03 20:40:20.930675 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.930580    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw" event={"ID":"0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9","Type":"ContainerDied","Data":"40d91fef7f4e493ef204d85865d192e8044938d4567cdc834a314fcfb8cb1262"}
Nov 03 20:40:20.961025 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.960785    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:20.974459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.974430    5659 scope.go:122] "RemoveContainer" containerID="9c5473d2d956c8eac1e6ed21263c2deaab3911e14804a6104c210aa0938f23c6"
Nov 03 20:40:20.974992 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:40:20.974959    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5473d2d956c8eac1e6ed21263c2deaab3911e14804a6104c210aa0938f23c6\": container with ID starting with 9c5473d2d956c8eac1e6ed21263c2deaab3911e14804a6104c210aa0938f23c6 not found: ID does not exist" containerID="9c5473d2d956c8eac1e6ed21263c2deaab3911e14804a6104c210aa0938f23c6"
Nov 03 20:40:20.975095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.975001    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5473d2d956c8eac1e6ed21263c2deaab3911e14804a6104c210aa0938f23c6"} err="failed to get container status \"9c5473d2d956c8eac1e6ed21263c2deaab3911e14804a6104c210aa0938f23c6\": rpc error: code = NotFound desc = could not find container \"9c5473d2d956c8eac1e6ed21263c2deaab3911e14804a6104c210aa0938f23c6\": container with ID starting with 9c5473d2d956c8eac1e6ed21263c2deaab3911e14804a6104c210aa0938f23c6 not found: ID does not exist"
Nov 03 20:40:20.975095 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.975027    5659 scope.go:122] "RemoveContainer" containerID="0ef467b5b183c95c0eb03b83b3425b091c627eb44f27af52a12462fb07499ef1"
Nov 03 20:40:20.985794 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.985769    5659 scope.go:122] "RemoveContainer" containerID="0ef467b5b183c95c0eb03b83b3425b091c627eb44f27af52a12462fb07499ef1"
Nov 03 20:40:20.986138 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:40:20.986108    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef467b5b183c95c0eb03b83b3425b091c627eb44f27af52a12462fb07499ef1\": container with ID starting with 0ef467b5b183c95c0eb03b83b3425b091c627eb44f27af52a12462fb07499ef1 not found: ID does not exist" containerID="0ef467b5b183c95c0eb03b83b3425b091c627eb44f27af52a12462fb07499ef1"
Nov 03 20:40:20.986264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.986146    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef467b5b183c95c0eb03b83b3425b091c627eb44f27af52a12462fb07499ef1"} err="failed to get container status \"0ef467b5b183c95c0eb03b83b3425b091c627eb44f27af52a12462fb07499ef1\": rpc error: code = NotFound desc = could not find container \"0ef467b5b183c95c0eb03b83b3425b091c627eb44f27af52a12462fb07499ef1\": container with ID starting with 0ef467b5b183c95c0eb03b83b3425b091c627eb44f27af52a12462fb07499ef1 not found: ID does not exist"
Nov 03 20:40:20.995429 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:20.995400    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"]
Nov 03 20:40:21.004435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.004398    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/distinct-w1-32e482-kserve-5b7d5b55db-6m4gf"]
Nov 03 20:40:21.004699 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.004674    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:21.004781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.004708    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92ffe4fd-3411-41b3-898f-44da3191ee57-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:21.004781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.004726    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92ffe4fd-3411-41b3-898f-44da3191ee57-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:21.004781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.004747    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:21.004781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.004762    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzdl8\" (UniqueName: \"kubernetes.io/projected/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-kube-api-access-bzdl8\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:21.004781 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.004777    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:21.005036 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.004795    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcldh\" (UniqueName: \"kubernetes.io/projected/92ffe4fd-3411-41b3-898f-44da3191ee57-kube-api-access-rcldh\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:21.015290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.015266    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"]
Nov 03 20:40:21.021626 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.021591    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/premium-w1-32e482-kserve-dd44fcc8c-jrcfw"]
Nov 03 20:40:21.139268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.139223    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:21.550674 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.550639    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:21.968884 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.968851    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" path="/var/lib/kubelet/pods/0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9/volumes"
Nov 03 20:40:21.969331 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:21.969317    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" path="/var/lib/kubelet/pods/92ffe4fd-3411-41b3-898f-44da3191ee57/volumes"
Nov 03 20:40:22.183859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:22.183811    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:22.683679 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:22.683626    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:23.016446 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.016404    5659 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:40:23.016681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.016465    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf" podUID="bee89175-06af-4e58-8b04-bd892b530f1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:40:23.039290 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.039241    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:23.207170 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.207126    5659 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:40:23.207343 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.207210    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" podUID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:40:23.488639 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.488612    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 20:40:23.637128 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.637042    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-envoy\") pod \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") "
Nov 03 20:40:23.637305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.637145    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-kube-api-access-qjtm8\" (UniqueName: \"kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-kube-api-access-qjtm8\") pod \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") "
Nov 03 20:40:23.637305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.637176    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-token\") pod \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") "
Nov 03 20:40:23.637305 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.637209    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-socket\") pod \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") "
Nov 03 20:40:23.637519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.637328    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-podinfo\") pod \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") "
Nov 03 20:40:23.637519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.637393    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-credential-socket\") pod \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") "
Nov 03 20:40:23.637519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.637461    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-data\") pod \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") "
Nov 03 20:40:23.637519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.637514    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istiod-ca-cert\") pod \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") "
Nov 03 20:40:23.637793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.637575    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-certs\") pod \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\" (UID: \"b1cb0655-e9f8-4d24-b524-5adbe643ba9d\") "
Nov 03 20:40:23.637909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.637869    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-data" pod "b1cb0655-e9f8-4d24-b524-5adbe643ba9d" (UID: "b1cb0655-e9f8-4d24-b524-5adbe643ba9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:23.638152 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.638123    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-certs" pod "b1cb0655-e9f8-4d24-b524-5adbe643ba9d" (UID: "b1cb0655-e9f8-4d24-b524-5adbe643ba9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:23.638236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.638131    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:23.638471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.638442    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istiod-ca-cert" pod "b1cb0655-e9f8-4d24-b524-5adbe643ba9d" (UID: "b1cb0655-e9f8-4d24-b524-5adbe643ba9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:23.638611 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.638529    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-credential-socket" pod "b1cb0655-e9f8-4d24-b524-5adbe643ba9d" (UID: "b1cb0655-e9f8-4d24-b524-5adbe643ba9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:23.638611 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.638580    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-socket" pod "b1cb0655-e9f8-4d24-b524-5adbe643ba9d" (UID: "b1cb0655-e9f8-4d24-b524-5adbe643ba9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:23.639713 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.639686    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-kube-api-access-qjtm8" pod "b1cb0655-e9f8-4d24-b524-5adbe643ba9d" (UID: "b1cb0655-e9f8-4d24-b524-5adbe643ba9d"). InnerVolumeSpecName "kube-api-access-qjtm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:23.639823 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.639738    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-envoy" pod "b1cb0655-e9f8-4d24-b524-5adbe643ba9d" (UID: "b1cb0655-e9f8-4d24-b524-5adbe643ba9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:23.640161 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.640122    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-podinfo" pod "b1cb0655-e9f8-4d24-b524-5adbe643ba9d" (UID: "b1cb0655-e9f8-4d24-b524-5adbe643ba9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:40:23.640269 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.640249    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-token" pod "b1cb0655-e9f8-4d24-b524-5adbe643ba9d" (UID: "b1cb0655-e9f8-4d24-b524-5adbe643ba9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:23.739526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.739491    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:23.739526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.739526    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjtm8\" (UniqueName: \"kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-kube-api-access-qjtm8\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:23.739983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.739540    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:23.739983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.739552    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:23.739983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.739565    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:23.739983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.739577    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:23.739983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.739594    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:23.739983 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.739607    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb0655-e9f8-4d24-b524-5adbe643ba9d-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:23.960816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.953369    5659 generic.go:356] "Generic (PLEG): container finished" podID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" containerID="668fd6d24c0fb31ea276a6cfe69f0c2ee33384ca7e5d57a6b4aaab31948857db" exitCode=0
Nov 03 20:40:23.960816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.953439    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"
Nov 03 20:40:23.960816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.953613    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" event={"ID":"b1cb0655-e9f8-4d24-b524-5adbe643ba9d","Type":"ContainerDied","Data":"668fd6d24c0fb31ea276a6cfe69f0c2ee33384ca7e5d57a6b4aaab31948857db"}
Nov 03 20:40:23.960816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.953668    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6" event={"ID":"b1cb0655-e9f8-4d24-b524-5adbe643ba9d","Type":"ContainerDied","Data":"2a27ca4f5454bc422fe51b2f500537f0f86b726ab7642e41e7aa8c5cd02f39cb"}
Nov 03 20:40:23.960816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.953700    5659 scope.go:122] "RemoveContainer" containerID="668fd6d24c0fb31ea276a6cfe69f0c2ee33384ca7e5d57a6b4aaab31948857db"
Nov 03 20:40:23.976299 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.972190    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:23.987631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.987601    5659 scope.go:122] "RemoveContainer" containerID="668fd6d24c0fb31ea276a6cfe69f0c2ee33384ca7e5d57a6b4aaab31948857db"
Nov 03 20:40:23.988648 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:40:23.988619    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668fd6d24c0fb31ea276a6cfe69f0c2ee33384ca7e5d57a6b4aaab31948857db\": container with ID starting with 668fd6d24c0fb31ea276a6cfe69f0c2ee33384ca7e5d57a6b4aaab31948857db not found: ID does not exist" containerID="668fd6d24c0fb31ea276a6cfe69f0c2ee33384ca7e5d57a6b4aaab31948857db"
Nov 03 20:40:23.988747 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:23.988659    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668fd6d24c0fb31ea276a6cfe69f0c2ee33384ca7e5d57a6b4aaab31948857db"} err="failed to get container status \"668fd6d24c0fb31ea276a6cfe69f0c2ee33384ca7e5d57a6b4aaab31948857db\": rpc error: code = NotFound desc = could not find container \"668fd6d24c0fb31ea276a6cfe69f0c2ee33384ca7e5d57a6b4aaab31948857db\": container with ID starting with 668fd6d24c0fb31ea276a6cfe69f0c2ee33384ca7e5d57a6b4aaab31948857db not found: ID does not exist"
Nov 03 20:40:24.000403 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:24.000370    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"]
Nov 03 20:40:24.009312 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:24.009286    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-667s6"]
Nov 03 20:40:24.651530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:24.651474    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:25.365937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:25.365906    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:25.972059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:25.972015    5659 generic.go:356] "Generic (PLEG): container finished" podID="bee89175-06af-4e58-8b04-bd892b530f1e" containerID="8b95a5948ef9b60fc35d20919ab0ad9f743cab22f5c1f57f89e65c64554c74e7" exitCode=0
Nov 03 20:40:25.972529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:25.972503    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" path="/var/lib/kubelet/pods/b1cb0655-e9f8-4d24-b524-5adbe643ba9d/volumes"
Nov 03 20:40:25.972875 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:25.972859    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"]
Nov 03 20:40:25.972930 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:25.972903    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf" event={"ID":"bee89175-06af-4e58-8b04-bd892b530f1e","Type":"ContainerDied","Data":"8b95a5948ef9b60fc35d20919ab0ad9f743cab22f5c1f57f89e65c64554c74e7"}
Nov 03 20:40:26.100559 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.100446    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:26.207820 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.207789    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:40:26.237730 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.236449    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:26.371621 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.371588    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee89175-06af-4e58-8b04-bd892b530f1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee89175-06af-4e58-8b04-bd892b530f1e-istiod-ca-cert\") pod \"bee89175-06af-4e58-8b04-bd892b530f1e\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") "
Nov 03 20:40:26.372144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.371655    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-socket\") pod \"bee89175-06af-4e58-8b04-bd892b530f1e\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") "
Nov 03 20:40:26.372144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.371691    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-credential-socket\") pod \"bee89175-06af-4e58-8b04-bd892b530f1e\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") "
Nov 03 20:40:26.372144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.371729    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-istio-token\" (UniqueName: \"kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-istio-token\") pod \"bee89175-06af-4e58-8b04-bd892b530f1e\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") "
Nov 03 20:40:26.372144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.371768    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-data\") pod \"bee89175-06af-4e58-8b04-bd892b530f1e\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") "
Nov 03 20:40:26.372144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.371865    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-envoy\") pod \"bee89175-06af-4e58-8b04-bd892b530f1e\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") "
Nov 03 20:40:26.372144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.371920    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-kube-api-access-94bsl\" (UniqueName: \"kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-kube-api-access-94bsl\") pod \"bee89175-06af-4e58-8b04-bd892b530f1e\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") "
Nov 03 20:40:26.372144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.371958    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-socket" pod "bee89175-06af-4e58-8b04-bd892b530f1e" (UID: "bee89175-06af-4e58-8b04-bd892b530f1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:26.372144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.372009    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-credential-socket" pod "bee89175-06af-4e58-8b04-bd892b530f1e" (UID: "bee89175-06af-4e58-8b04-bd892b530f1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:26.372144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.372022    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bee89175-06af-4e58-8b04-bd892b530f1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee89175-06af-4e58-8b04-bd892b530f1e-istio-podinfo\") pod \"bee89175-06af-4e58-8b04-bd892b530f1e\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") "
Nov 03 20:40:26.372144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.372090    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-certs\") pod \"bee89175-06af-4e58-8b04-bd892b530f1e\" (UID: \"bee89175-06af-4e58-8b04-bd892b530f1e\") "
Nov 03 20:40:26.372144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.372096    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee89175-06af-4e58-8b04-bd892b530f1e-istiod-ca-cert" pod "bee89175-06af-4e58-8b04-bd892b530f1e" (UID: "bee89175-06af-4e58-8b04-bd892b530f1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:26.372805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.372558    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-data" pod "bee89175-06af-4e58-8b04-bd892b530f1e" (UID: "bee89175-06af-4e58-8b04-bd892b530f1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:26.372805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.372572    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee89175-06af-4e58-8b04-bd892b530f1e-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:26.372805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.372596    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:26.372805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.372620    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:26.372805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.372593    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-certs" pod "bee89175-06af-4e58-8b04-bd892b530f1e" (UID: "bee89175-06af-4e58-8b04-bd892b530f1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:26.374465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.374430    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-kube-api-access-94bsl" pod "bee89175-06af-4e58-8b04-bd892b530f1e" (UID: "bee89175-06af-4e58-8b04-bd892b530f1e"). InnerVolumeSpecName "kube-api-access-94bsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:26.374716 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.374690    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-envoy" pod "bee89175-06af-4e58-8b04-bd892b530f1e" (UID: "bee89175-06af-4e58-8b04-bd892b530f1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:26.374839 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.374810    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-istio-token" pod "bee89175-06af-4e58-8b04-bd892b530f1e" (UID: "bee89175-06af-4e58-8b04-bd892b530f1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:26.374966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.374946    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bee89175-06af-4e58-8b04-bd892b530f1e-istio-podinfo" pod "bee89175-06af-4e58-8b04-bd892b530f1e" (UID: "bee89175-06af-4e58-8b04-bd892b530f1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:40:26.474311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.474267    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee89175-06af-4e58-8b04-bd892b530f1e-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:26.474311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.474302    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:26.474311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.474315    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:26.474606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.474328    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:26.474606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.474340    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee89175-06af-4e58-8b04-bd892b530f1e-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:26.474606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.474352    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94bsl\" (UniqueName: \"kubernetes.io/projected/bee89175-06af-4e58-8b04-bd892b530f1e-kube-api-access-94bsl\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:26.981720 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.981688    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"
Nov 03 20:40:26.981909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.981715    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf" event={"ID":"bee89175-06af-4e58-8b04-bd892b530f1e","Type":"ContainerDied","Data":"60d196ac4d6ac26a2b01fdbd9a19273819d066c6fec7abc8294bf8c12fcd5892"}
Nov 03 20:40:26.981909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:26.981775    5659 scope.go:122] "RemoveContainer" containerID="8b95a5948ef9b60fc35d20919ab0ad9f743cab22f5c1f57f89e65c64554c74e7"
Nov 03 20:40:27.012951 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:27.012916    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"]
Nov 03 20:40:27.018880 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:27.018849    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-fgtxf"]
Nov 03 20:40:27.044224 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:27.044185    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:27.702252 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:27.702214    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:27.968942 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:27.968910    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee89175-06af-4e58-8b04-bd892b530f1e" path="/var/lib/kubelet/pods/bee89175-06af-4e58-8b04-bd892b530f1e/volumes"
Nov 03 20:40:28.140921 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:28.140882    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:28.332036 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:28.331939    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:28.472831 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:28.472793    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:28.941427 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:28.941390    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:29.075890 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:29.075859    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:29.739990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:29.739953    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:30.414472 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:30.414438    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:30.567620 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:30.567587    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:31.498369 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:31.498334    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:32.259165 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:32.259128    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:32.393436 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:32.393380    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:32.558667 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:32.558576    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:33.160476 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:33.159910    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:33.844918 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:33.844883    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:34.165519 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:34.165372    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:34.513448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:34.513412    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:34.723979 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:34.723942    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:35.340717 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:35.340681    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:36.134541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:36.134508    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:36.960908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:36.959766    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:37.814507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:37.810246    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:38.518217 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:38.518184    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:39.265941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:39.265174    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:39.599280 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:39.599196    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:39.753407 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:39.753373    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:40.257610 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:40.257573    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:40.945692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:40.945660    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:40.995264 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:40.995218    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main" containerID="cri-o://90cf9d1e6a36dacb86298d9be5e5cf8cfcb1f4c6cf602fa95ff7bb47cb5de4d3" gracePeriod=45
Nov 03 20:40:41.077093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.076848    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:41.256973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.256929    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:40:41.300649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.298192    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-dshm\") pod \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") "
Nov 03 20:40:41.300649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.298461    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tmp-dir\") pod \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") "
Nov 03 20:40:41.300649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.298541    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-model-cache\") pod \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") "
Nov 03 20:40:41.300649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.298869    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-home\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-home\") pod \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") "
Nov 03 20:40:41.300649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.298943    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tmp-dir" pod "11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" (UID: "11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.300649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.299146    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-model-cache" pod "11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" (UID: "11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.300649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.299219    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-home" pod "11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" (UID: "11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.300649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.299294    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-kube-api-access-sxgmw\" (UniqueName: \"kubernetes.io/projected/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-kube-api-access-sxgmw\") pod \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") "
Nov 03 20:40:41.300649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.299395    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tls-certs\") pod \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\" (UID: \"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a\") "
Nov 03 20:40:41.300649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.299881    5659 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-home\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.300649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.299903    5659 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tmp-dir\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.300649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.299919    5659 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-model-cache\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.313509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.312629    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-dshm" pod "11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" (UID: "11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.313509 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.312668    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-kube-api-access-sxgmw" pod "11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" (UID: "11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a"). InnerVolumeSpecName "kube-api-access-sxgmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:41.314640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.314610    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tls-certs" pod "11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" (UID: "11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:41.401385 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.401351    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgmw\" (UniqueName: \"kubernetes.io/projected/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-kube-api-access-sxgmw\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.401385 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.401384    5659 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-tls-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.401615 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.401396    5659 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a-dshm\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.872214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:41.872181    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:42.032763 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.032728    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:42.109598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.109556    5659 generic.go:356] "Generic (PLEG): container finished" podID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerID="90cf9d1e6a36dacb86298d9be5e5cf8cfcb1f4c6cf602fa95ff7bb47cb5de4d3" exitCode=0
Nov 03 20:40:42.109809 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.109787    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" event={"ID":"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a","Type":"ContainerDied","Data":"90cf9d1e6a36dacb86298d9be5e5cf8cfcb1f4c6cf602fa95ff7bb47cb5de4d3"}
Nov 03 20:40:42.109877 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.109829    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr" event={"ID":"11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a","Type":"ContainerDied","Data":"eee72930bdba7d583eb95c29121c4c39261536fabecba7d1871319f9f910a2dc"}
Nov 03 20:40:42.109877 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.109852    5659 scope.go:122] "RemoveContainer" containerID="90cf9d1e6a36dacb86298d9be5e5cf8cfcb1f4c6cf602fa95ff7bb47cb5de4d3"
Nov 03 20:40:42.110059 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.110039    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"
Nov 03 20:40:42.123411 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.123389    5659 scope.go:122] "RemoveContainer" containerID="90cf9d1e6a36dacb86298d9be5e5cf8cfcb1f4c6cf602fa95ff7bb47cb5de4d3"
Nov 03 20:40:42.123808 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:40:42.123780    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90cf9d1e6a36dacb86298d9be5e5cf8cfcb1f4c6cf602fa95ff7bb47cb5de4d3\": container with ID starting with 90cf9d1e6a36dacb86298d9be5e5cf8cfcb1f4c6cf602fa95ff7bb47cb5de4d3 not found: ID does not exist" containerID="90cf9d1e6a36dacb86298d9be5e5cf8cfcb1f4c6cf602fa95ff7bb47cb5de4d3"
Nov 03 20:40:42.123933 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.123819    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90cf9d1e6a36dacb86298d9be5e5cf8cfcb1f4c6cf602fa95ff7bb47cb5de4d3"} err="failed to get container status \"90cf9d1e6a36dacb86298d9be5e5cf8cfcb1f4c6cf602fa95ff7bb47cb5de4d3\": rpc error: code = NotFound desc = could not find container \"90cf9d1e6a36dacb86298d9be5e5cf8cfcb1f4c6cf602fa95ff7bb47cb5de4d3\": container with ID starting with 90cf9d1e6a36dacb86298d9be5e5cf8cfcb1f4c6cf602fa95ff7bb47cb5de4d3 not found: ID does not exist"
Nov 03 20:40:42.137532 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.137478    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"]
Nov 03 20:40:42.147931 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.147900    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/distinct-w3-9f46e5-kserve-bd48c4f5c-v4dgr"]
Nov 03 20:40:42.240599 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.240561    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:42.470677 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.464932    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:42.669426 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.669389    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:42.799261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:42.799176    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:43.334250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:43.334219    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:43.968536 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:43.968461    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" path="/var/lib/kubelet/pods/11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a/volumes"
Nov 03 20:40:44.052899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:44.052864    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:44.344044 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:44.343961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:44.635989 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:44.635890    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:44.873952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:44.873901    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"]
Nov 03 20:40:44.875984 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:44.875948    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" podUID="b793166f-3de9-4167-97d6-e3cd33c7a72f" containerName="istio-proxy" containerID="cri-o://4a9542b6b060da80a0a3933b260da69c7ddc74b5866bcb5c6d4b51a16fc70c9a" gracePeriod=30
Nov 03 20:40:44.936381 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:44.936307    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:45.069546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:45.068261    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:45.217122 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:40:45.217079    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:40:46.076583 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:46.076545    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:46.866855 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:46.866396    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:47.557926 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:47.557829    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:48.041899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:48.041851    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:48.237149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:48.237097    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:48.819472 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:48.819435    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:48.958898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:48.958856    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:49.637591 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:49.637554    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:50.161719 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.161652    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 20:40:50.165730 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.165699    5659 generic.go:356] "Generic (PLEG): container finished" podID="b793166f-3de9-4167-97d6-e3cd33c7a72f" containerID="4a9542b6b060da80a0a3933b260da69c7ddc74b5866bcb5c6d4b51a16fc70c9a" exitCode=0
Nov 03 20:40:50.165864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.165736    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"
Nov 03 20:40:50.165957 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.165866    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" event={"ID":"b793166f-3de9-4167-97d6-e3cd33c7a72f","Type":"ContainerDied","Data":"4a9542b6b060da80a0a3933b260da69c7ddc74b5866bcb5c6d4b51a16fc70c9a"}
Nov 03 20:40:50.165957 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.165905    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz" event={"ID":"b793166f-3de9-4167-97d6-e3cd33c7a72f","Type":"ContainerDied","Data":"652ce74cc3e10b06ee1889b99d4e2618f7b6cfdae752d1ed918413800dca6a4c"}
Nov 03 20:40:50.165957 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.165934    5659 scope.go:122] "RemoveContainer" containerID="4a9542b6b060da80a0a3933b260da69c7ddc74b5866bcb5c6d4b51a16fc70c9a"
Nov 03 20:40:50.177815 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.177793    5659 scope.go:122] "RemoveContainer" containerID="4a9542b6b060da80a0a3933b260da69c7ddc74b5866bcb5c6d4b51a16fc70c9a"
Nov 03 20:40:50.178076 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:40:50.178057    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9542b6b060da80a0a3933b260da69c7ddc74b5866bcb5c6d4b51a16fc70c9a\": container with ID starting with 4a9542b6b060da80a0a3933b260da69c7ddc74b5866bcb5c6d4b51a16fc70c9a not found: ID does not exist" containerID="4a9542b6b060da80a0a3933b260da69c7ddc74b5866bcb5c6d4b51a16fc70c9a"
Nov 03 20:40:50.178164 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.178084    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9542b6b060da80a0a3933b260da69c7ddc74b5866bcb5c6d4b51a16fc70c9a"} err="failed to get container status \"4a9542b6b060da80a0a3933b260da69c7ddc74b5866bcb5c6d4b51a16fc70c9a\": rpc error: code = NotFound desc = could not find container \"4a9542b6b060da80a0a3933b260da69c7ddc74b5866bcb5c6d4b51a16fc70c9a\": container with ID starting with 4a9542b6b060da80a0a3933b260da69c7ddc74b5866bcb5c6d4b51a16fc70c9a not found: ID does not exist"
Nov 03 20:40:50.235940 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.235910    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-token\" (UniqueName: \"kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-token\") pod \"b793166f-3de9-4167-97d6-e3cd33c7a72f\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") "
Nov 03 20:40:50.236109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.235983    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-socket\") pod \"b793166f-3de9-4167-97d6-e3cd33c7a72f\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") "
Nov 03 20:40:50.236109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236014    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-envoy\") pod \"b793166f-3de9-4167-97d6-e3cd33c7a72f\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") "
Nov 03 20:40:50.236109 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236087    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-podinfo\") pod \"b793166f-3de9-4167-97d6-e3cd33c7a72f\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") "
Nov 03 20:40:50.236252 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236157    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-certs\") pod \"b793166f-3de9-4167-97d6-e3cd33c7a72f\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") "
Nov 03 20:40:50.236252 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236233    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-kube-api-access-jvn7s\" (UniqueName: \"kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-kube-api-access-jvn7s\") pod \"b793166f-3de9-4167-97d6-e3cd33c7a72f\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") "
Nov 03 20:40:50.236360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236274    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b793166f-3de9-4167-97d6-e3cd33c7a72f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793166f-3de9-4167-97d6-e3cd33c7a72f-istiod-ca-cert\") pod \"b793166f-3de9-4167-97d6-e3cd33c7a72f\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") "
Nov 03 20:40:50.236360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236306    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-credential-socket\") pod \"b793166f-3de9-4167-97d6-e3cd33c7a72f\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") "
Nov 03 20:40:50.236360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236330    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-socket" pod "b793166f-3de9-4167-97d6-e3cd33c7a72f" (UID: "b793166f-3de9-4167-97d6-e3cd33c7a72f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.236360 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236348    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-data\") pod \"b793166f-3de9-4167-97d6-e3cd33c7a72f\" (UID: \"b793166f-3de9-4167-97d6-e3cd33c7a72f\") "
Nov 03 20:40:50.236617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236580    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-credential-socket" pod "b793166f-3de9-4167-97d6-e3cd33c7a72f" (UID: "b793166f-3de9-4167-97d6-e3cd33c7a72f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.236919 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236716    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-certs" pod "b793166f-3de9-4167-97d6-e3cd33c7a72f" (UID: "b793166f-3de9-4167-97d6-e3cd33c7a72f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.237086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236774    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b793166f-3de9-4167-97d6-e3cd33c7a72f-istiod-ca-cert" pod "b793166f-3de9-4167-97d6-e3cd33c7a72f" (UID: "b793166f-3de9-4167-97d6-e3cd33c7a72f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:50.237086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236796    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-data" pod "b793166f-3de9-4167-97d6-e3cd33c7a72f" (UID: "b793166f-3de9-4167-97d6-e3cd33c7a72f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.237086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236849    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.237086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.236964    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.238706 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.238677    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-kube-api-access-jvn7s" pod "b793166f-3de9-4167-97d6-e3cd33c7a72f" (UID: "b793166f-3de9-4167-97d6-e3cd33c7a72f"). InnerVolumeSpecName "kube-api-access-jvn7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:50.238859 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.238814    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-token" pod "b793166f-3de9-4167-97d6-e3cd33c7a72f" (UID: "b793166f-3de9-4167-97d6-e3cd33c7a72f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:50.238962 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.238943    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-envoy" pod "b793166f-3de9-4167-97d6-e3cd33c7a72f" (UID: "b793166f-3de9-4167-97d6-e3cd33c7a72f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.239016 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.238985    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-podinfo" pod "b793166f-3de9-4167-97d6-e3cd33c7a72f" (UID: "b793166f-3de9-4167-97d6-e3cd33c7a72f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:40:50.337734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.337704    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.337734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.337731    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.337734 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.337743    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.338006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.337752    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvn7s\" (UniqueName: \"kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-kube-api-access-jvn7s\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.338006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.337763    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793166f-3de9-4167-97d6-e3cd33c7a72f-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.338006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.337771    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.338006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.337781    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b793166f-3de9-4167-97d6-e3cd33c7a72f-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.351458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.351428    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:50.495966 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.495930    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"]
Nov 03 20:40:50.501435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.501407    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-rrvsz"]
Nov 03 20:40:50.725149 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.725116    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:50.882061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:50.881977    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:51.343144 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:51.343105    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:51.923409 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:51.923358    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:52.008699 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:52.008071    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b793166f-3de9-4167-97d6-e3cd33c7a72f" path="/var/lib/kubelet/pods/b793166f-3de9-4167-97d6-e3cd33c7a72f/volumes"
Nov 03 20:40:52.060231 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:52.060195    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:52.187421 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:52.187339    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:52.413492 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:52.413443    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"]
Nov 03 20:40:52.413974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:52.413686    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy" containerID="cri-o://9816c0426f3f554e6a5570384a178e4b4e503a4200e01dcc01469240c76b3a5f" gracePeriod=30
Nov 03 20:40:52.466816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:52.466781    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:53.921539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:53.921508    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:54.050339 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:54.050292    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:56.979984 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:56.979947    5659 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:40:56.980400 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:56.980008    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:40:57.683295 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.683218    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 20:40:57.831696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.831661    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-kube-api-access-k8pgr\" (UniqueName: \"kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-kube-api-access-k8pgr\") pod \"40c95cd0-cf66-4a36-9c3e-f917ee445790\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") "
Nov 03 20:40:57.831696 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.831706    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-podinfo\") pod \"40c95cd0-cf66-4a36-9c3e-f917ee445790\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") "
Nov 03 20:40:57.831988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.831768    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-data\") pod \"40c95cd0-cf66-4a36-9c3e-f917ee445790\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") "
Nov 03 20:40:57.831988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.831816    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-envoy\") pod \"40c95cd0-cf66-4a36-9c3e-f917ee445790\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") "
Nov 03 20:40:57.831988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.831909    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-credential-socket\") pod \"40c95cd0-cf66-4a36-9c3e-f917ee445790\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") "
Nov 03 20:40:57.831988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.831943    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-token\" (UniqueName: \"kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-token\") pod \"40c95cd0-cf66-4a36-9c3e-f917ee445790\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") "
Nov 03 20:40:57.831988 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.831975    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c95cd0-cf66-4a36-9c3e-f917ee445790-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c95cd0-cf66-4a36-9c3e-f917ee445790-istiod-ca-cert\") pod \"40c95cd0-cf66-4a36-9c3e-f917ee445790\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") "
Nov 03 20:40:57.832287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.832013    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-certs\") pod \"40c95cd0-cf66-4a36-9c3e-f917ee445790\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") "
Nov 03 20:40:57.832287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.832083    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-data" pod "40c95cd0-cf66-4a36-9c3e-f917ee445790" (UID: "40c95cd0-cf66-4a36-9c3e-f917ee445790"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.832287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.832096    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-socket\") pod \"40c95cd0-cf66-4a36-9c3e-f917ee445790\" (UID: \"40c95cd0-cf66-4a36-9c3e-f917ee445790\") "
Nov 03 20:40:57.832287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.832234    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-credential-socket" pod "40c95cd0-cf66-4a36-9c3e-f917ee445790" (UID: "40c95cd0-cf66-4a36-9c3e-f917ee445790"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.832575 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.832336    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-socket" pod "40c95cd0-cf66-4a36-9c3e-f917ee445790" (UID: "40c95cd0-cf66-4a36-9c3e-f917ee445790"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.832575 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.832562    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-certs" pod "40c95cd0-cf66-4a36-9c3e-f917ee445790" (UID: "40c95cd0-cf66-4a36-9c3e-f917ee445790"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.832970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.832699    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.832970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.832725    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.832970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.832741    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.832970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.832755    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.832970 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.832939    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c95cd0-cf66-4a36-9c3e-f917ee445790-istiod-ca-cert" pod "40c95cd0-cf66-4a36-9c3e-f917ee445790" (UID: "40c95cd0-cf66-4a36-9c3e-f917ee445790"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:57.834530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.834504    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-kube-api-access-k8pgr" pod "40c95cd0-cf66-4a36-9c3e-f917ee445790" (UID: "40c95cd0-cf66-4a36-9c3e-f917ee445790"). InnerVolumeSpecName "kube-api-access-k8pgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:57.834729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.834710    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-podinfo" pod "40c95cd0-cf66-4a36-9c3e-f917ee445790" (UID: "40c95cd0-cf66-4a36-9c3e-f917ee445790"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:40:57.834924 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.834903    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-envoy" pod "40c95cd0-cf66-4a36-9c3e-f917ee445790" (UID: "40c95cd0-cf66-4a36-9c3e-f917ee445790"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.835000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.834918    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-token" pod "40c95cd0-cf66-4a36-9c3e-f917ee445790" (UID: "40c95cd0-cf66-4a36-9c3e-f917ee445790"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:57.933994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.933914    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8pgr\" (UniqueName: \"kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-kube-api-access-k8pgr\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.933994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.933964    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.933994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.933974    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.933994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.933983    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c95cd0-cf66-4a36-9c3e-f917ee445790-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.933994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:57.933993    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c95cd0-cf66-4a36-9c3e-f917ee445790-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:58.223695 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:58.223657    5659 generic.go:356] "Generic (PLEG): container finished" podID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerID="9816c0426f3f554e6a5570384a178e4b4e503a4200e01dcc01469240c76b3a5f" exitCode=0
Nov 03 20:40:58.224203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:58.223707    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"
Nov 03 20:40:58.224203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:58.223714    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" event={"ID":"40c95cd0-cf66-4a36-9c3e-f917ee445790","Type":"ContainerDied","Data":"9816c0426f3f554e6a5570384a178e4b4e503a4200e01dcc01469240c76b3a5f"}
Nov 03 20:40:58.224203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:58.223815    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h" event={"ID":"40c95cd0-cf66-4a36-9c3e-f917ee445790","Type":"ContainerDied","Data":"c3022ef8cab9bf20e3ebb931a37fb4b95e0158df49f49a880ea0144a33cf59f2"}
Nov 03 20:40:58.224203 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:58.223836    5659 scope.go:122] "RemoveContainer" containerID="9816c0426f3f554e6a5570384a178e4b4e503a4200e01dcc01469240c76b3a5f"
Nov 03 20:40:58.234060 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:58.234030    5659 scope.go:122] "RemoveContainer" containerID="9816c0426f3f554e6a5570384a178e4b4e503a4200e01dcc01469240c76b3a5f"
Nov 03 20:40:58.234328 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:40:58.234309    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9816c0426f3f554e6a5570384a178e4b4e503a4200e01dcc01469240c76b3a5f\": container with ID starting with 9816c0426f3f554e6a5570384a178e4b4e503a4200e01dcc01469240c76b3a5f not found: ID does not exist" containerID="9816c0426f3f554e6a5570384a178e4b4e503a4200e01dcc01469240c76b3a5f"
Nov 03 20:40:58.234393 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:58.234338    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9816c0426f3f554e6a5570384a178e4b4e503a4200e01dcc01469240c76b3a5f"} err="failed to get container status \"9816c0426f3f554e6a5570384a178e4b4e503a4200e01dcc01469240c76b3a5f\": rpc error: code = NotFound desc = could not find container \"9816c0426f3f554e6a5570384a178e4b4e503a4200e01dcc01469240c76b3a5f\": container with ID starting with 9816c0426f3f554e6a5570384a178e4b4e503a4200e01dcc01469240c76b3a5f not found: ID does not exist"
Nov 03 20:40:58.246448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:58.246420    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"]
Nov 03 20:40:58.251338 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:58.251305    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-4pz9h"]
Nov 03 20:40:59.969756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:40:59.969721    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" path="/var/lib/kubelet/pods/40c95cd0-cf66-4a36-9c3e-f917ee445790/volumes"
Nov 03 20:41:03.669978 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:03.669936    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:03.807049 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:03.806966    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:03.943997 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:03.943905    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:05.074808 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:05.074777    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:06.575408 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:06.575363    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:10.519821 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:10.519788    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:10.665330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:10.665289    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:10.994422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:10.994313    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:11.303631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:11.303543    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"]
Nov 03 20:41:11.304185 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:11.304129    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2" podUID="fdde0ea7-99e2-4098-8e03-db9b8533aab9" containerName="istio-proxy" containerID="cri-o://a6c5fdf52913cd2217a9fac969383336ded73621f32982049e55134458f8cc6a" gracePeriod=30
Nov 03 20:41:11.354448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:11.354412    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:12.252841 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:12.252805    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66b59d468d-j8ndl_19730c58-ac26-42dc-bd3a-8ec0f6076264/authorino/0.log"
Nov 03 20:41:12.676673 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:12.676583    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pmwn5_a8606f49-1dc0-45c2-9720-be9731a089c4/manager/0.log"
Nov 03 20:41:12.842617 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:12.842586    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-t4sb5_856603f3-24e3-4d9e-b6dc-68dc3f57d18a/kuadrant-console-plugin/0.log"
Nov 03 20:41:13.224681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:13.224637    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-l8fcr_06729d18-3d24-489f-9e57-4b24a6ca6323/manager/0.log"
Nov 03 20:41:15.016654 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:15.016621    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fxxs9_aa079182-95a7-4454-be04-45f06a41088f/manager/0.log"
Nov 03 20:41:15.217115 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:41:15.217068    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:41:15.238090 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:15.238056    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qkjdd_950e6c44-5e74-4c6c-9c38-0712a81af85a/manager/0.log"
Nov 03 20:41:16.293733 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.293700    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-76gw5_0f42d32f-c8f1-48b4-807a-76636151cb72/server/0.log"
Nov 03 20:41:16.565422 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.565388    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 20:41:16.648731 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.648692    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-credential-socket\") pod \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") "
Nov 03 20:41:16.648911 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.648750    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-certs\") pod \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") "
Nov 03 20:41:16.648911 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.648771    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istiod-ca-cert\") pod \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") "
Nov 03 20:41:16.648911 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.648796    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-kube-api-access-qchbk\" (UniqueName: \"kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-kube-api-access-qchbk\") pod \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") "
Nov 03 20:41:16.648911 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.648873    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-envoy\") pod \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") "
Nov 03 20:41:16.649115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.648983    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-socket\") pod \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") "
Nov 03 20:41:16.649115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.649031    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-token\" (UniqueName: \"kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-token\") pod \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") "
Nov 03 20:41:16.649115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.649048    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-credential-socket" pod "fdde0ea7-99e2-4098-8e03-db9b8533aab9" (UID: "fdde0ea7-99e2-4098-8e03-db9b8533aab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:16.649115 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.649076    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-certs" pod "fdde0ea7-99e2-4098-8e03-db9b8533aab9" (UID: "fdde0ea7-99e2-4098-8e03-db9b8533aab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:16.649392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.649133    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-data\") pod \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") "
Nov 03 20:41:16.649392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.649168    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-podinfo\") pod \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\" (UID: \"fdde0ea7-99e2-4098-8e03-db9b8533aab9\") "
Nov 03 20:41:16.649392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.649346    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-socket" pod "fdde0ea7-99e2-4098-8e03-db9b8533aab9" (UID: "fdde0ea7-99e2-4098-8e03-db9b8533aab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:16.649587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.649401    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-data" pod "fdde0ea7-99e2-4098-8e03-db9b8533aab9" (UID: "fdde0ea7-99e2-4098-8e03-db9b8533aab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:16.649587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.649449    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istiod-ca-cert" pod "fdde0ea7-99e2-4098-8e03-db9b8533aab9" (UID: "fdde0ea7-99e2-4098-8e03-db9b8533aab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:41:16.649722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.649692    5659 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.649722 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.649717    5659 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istiod-ca-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.649930 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.649732    5659 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-workload-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.649930 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.649746    5659 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-data\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.649930 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.649762    5659 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-credential-socket\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.651262 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.651227    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-envoy" pod "fdde0ea7-99e2-4098-8e03-db9b8533aab9" (UID: "fdde0ea7-99e2-4098-8e03-db9b8533aab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:16.651377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.651314    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-token" pod "fdde0ea7-99e2-4098-8e03-db9b8533aab9" (UID: "fdde0ea7-99e2-4098-8e03-db9b8533aab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:16.651377 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.651357    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-kube-api-access-qchbk" pod "fdde0ea7-99e2-4098-8e03-db9b8533aab9" (UID: "fdde0ea7-99e2-4098-8e03-db9b8533aab9"). InnerVolumeSpecName "kube-api-access-qchbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:16.651472 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.651382    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-podinfo" pod "fdde0ea7-99e2-4098-8e03-db9b8533aab9" (UID: "fdde0ea7-99e2-4098-8e03-db9b8533aab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:41:16.737146 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.737116    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-564sz_67be1eda-e12d-4a81-9785-5c5c8c3989d2/proxy/0.log"
Nov 03 20:41:16.750782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.750757    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.750782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.750781    5659 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-podinfo\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.750904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.750792    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qchbk\" (UniqueName: \"kubernetes.io/projected/fdde0ea7-99e2-4098-8e03-db9b8533aab9-kube-api-access-qchbk\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.750904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:16.750804    5659 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdde0ea7-99e2-4098-8e03-db9b8533aab9-istio-envoy\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:17.346441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:17.346402    5659 generic.go:356] "Generic (PLEG): container finished" podID="fdde0ea7-99e2-4098-8e03-db9b8533aab9" containerID="a6c5fdf52913cd2217a9fac969383336ded73621f32982049e55134458f8cc6a" exitCode=0
Nov 03 20:41:17.346899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:17.346455    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"
Nov 03 20:41:17.346899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:17.346496    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2" event={"ID":"fdde0ea7-99e2-4098-8e03-db9b8533aab9","Type":"ContainerDied","Data":"a6c5fdf52913cd2217a9fac969383336ded73621f32982049e55134458f8cc6a"}
Nov 03 20:41:17.346899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:17.346556    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2" event={"ID":"fdde0ea7-99e2-4098-8e03-db9b8533aab9","Type":"ContainerDied","Data":"f88c24c7100bc819bd63268788cbd82ddd1107c1fa142d4d40306eb3253ca486"}
Nov 03 20:41:17.346899 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:17.346578    5659 scope.go:122] "RemoveContainer" containerID="a6c5fdf52913cd2217a9fac969383336ded73621f32982049e55134458f8cc6a"
Nov 03 20:41:17.357648 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:17.357628    5659 scope.go:122] "RemoveContainer" containerID="a6c5fdf52913cd2217a9fac969383336ded73621f32982049e55134458f8cc6a"
Nov 03 20:41:17.357920 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:41:17.357895    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c5fdf52913cd2217a9fac969383336ded73621f32982049e55134458f8cc6a\": container with ID starting with a6c5fdf52913cd2217a9fac969383336ded73621f32982049e55134458f8cc6a not found: ID does not exist" containerID="a6c5fdf52913cd2217a9fac969383336ded73621f32982049e55134458f8cc6a"
Nov 03 20:41:17.357991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:17.357929    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c5fdf52913cd2217a9fac969383336ded73621f32982049e55134458f8cc6a"} err="failed to get container status \"a6c5fdf52913cd2217a9fac969383336ded73621f32982049e55134458f8cc6a\": rpc error: code = NotFound desc = could not find container \"a6c5fdf52913cd2217a9fac969383336ded73621f32982049e55134458f8cc6a\": container with ID starting with a6c5fdf52913cd2217a9fac969383336ded73621f32982049e55134458f8cc6a not found: ID does not exist"
Nov 03 20:41:17.372460 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:17.372428    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"]
Nov 03 20:41:17.376789 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:17.376766    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-6zqs2"]
Nov 03 20:41:17.969646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:17.969611    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdde0ea7-99e2-4098-8e03-db9b8533aab9" path="/var/lib/kubelet/pods/fdde0ea7-99e2-4098-8e03-db9b8533aab9/volumes"
Nov 03 20:41:29.888050 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:29.888017    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:30.030112 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:30.030078    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:30.833183 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:30.833151    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:30.966816 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:30.966785    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:41.259531 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:41.259465    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:42.836323 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:42.836278    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:44.225309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:44.225278    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:44.355546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:44.355503    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:45.216370 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:41:45.216332    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:41:45.454877 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:45.454840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:45.590793 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:45.590666    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:46.255866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:46.255827    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:47.087680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:47.087633    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:47.846208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:47.846166    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:48.728355 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:48.728323    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:49.359529 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:49.359469    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:50.031690 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:50.031657    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:50.764237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:50.764151    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:51.447741 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:51.447703    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:52.133027 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:52.132988    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:52.846080 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:52.846046    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:53.695237 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:53.695201    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:54.431383 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:54.430997    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:55.208542 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:55.208511    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:55.958324 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:55.958185    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:56.653283 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:56.651039    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:57.541929 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:57.541888    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:58.238108 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:58.238078    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:58.934967 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:58.934925    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:59.638018 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:41:59.637982    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:00.347471 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:00.347422    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:01.055512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:01.055438    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:01.836439 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:01.836402    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:02.547337 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:02.547300    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:03.254663 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:03.254616    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:03.950043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:03.950007    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:04.656179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:04.656135    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:05.414910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:05.414874    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:05.540825 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:05.540788    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:05.979089 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:05.979044    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:06.133866 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:06.133830    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:06.269330 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:06.269230    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:06.642493 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:06.642400    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:11.527933 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:11.527840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:13.901949 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:13.901904    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:15.219006 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:42:15.218966    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:42:21.583287 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:21.583243    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:23.886865 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:23.886827    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"]
Nov 03 20:42:23.887403 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:23.887045    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r" podUID="f76f8240-ab05-410c-8a60-1d7b8cdde912" containerName="discovery" containerID="cri-o://967f7fd677ea9e5d84ce8df0fd78ecbf8199f16e64bd2c017a91c1ba2a59b49b" gracePeriod=30
Nov 03 20:42:24.141974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.141913    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:42:24.288921 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.288882    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76f8240-ab05-410c-8a60-1d7b8cdde912-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76f8240-ab05-410c-8a60-1d7b8cdde912-local-certs\") pod \"f76f8240-ab05-410c-8a60-1d7b8cdde912\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") "
Nov 03 20:42:24.289121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.288938    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-token\" (UniqueName: \"kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-token\") pod \"f76f8240-ab05-410c-8a60-1d7b8cdde912\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") "
Nov 03 20:42:24.289121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.288983    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-kube-api-access-wvzpv\" (UniqueName: \"kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-kube-api-access-wvzpv\") pod \"f76f8240-ab05-410c-8a60-1d7b8cdde912\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") "
Nov 03 20:42:24.289121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.289075    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-ca-configmap\") pod \"f76f8240-ab05-410c-8a60-1d7b8cdde912\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") "
Nov 03 20:42:24.289121 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.289101    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-dns-cert\") pod \"f76f8240-ab05-410c-8a60-1d7b8cdde912\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") "
Nov 03 20:42:24.289363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.289138    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-cacerts\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-cacerts\") pod \"f76f8240-ab05-410c-8a60-1d7b8cdde912\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") "
Nov 03 20:42:24.289363 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.289186    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-kubeconfig\") pod \"f76f8240-ab05-410c-8a60-1d7b8cdde912\" (UID: \"f76f8240-ab05-410c-8a60-1d7b8cdde912\") "
Nov 03 20:42:24.289563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.289520    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-ca-configmap" pod "f76f8240-ab05-410c-8a60-1d7b8cdde912" (UID: "f76f8240-ab05-410c-8a60-1d7b8cdde912"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:42:24.291845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.291787    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-kubeconfig" pod "f76f8240-ab05-410c-8a60-1d7b8cdde912" (UID: "f76f8240-ab05-410c-8a60-1d7b8cdde912"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:42:24.291952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.291867    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76f8240-ab05-410c-8a60-1d7b8cdde912-local-certs" pod "f76f8240-ab05-410c-8a60-1d7b8cdde912" (UID: "f76f8240-ab05-410c-8a60-1d7b8cdde912"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:42:24.291952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.291872    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-cacerts" pod "f76f8240-ab05-410c-8a60-1d7b8cdde912" (UID: "f76f8240-ab05-410c-8a60-1d7b8cdde912"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:42:24.291952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.291895    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-dns-cert" pod "f76f8240-ab05-410c-8a60-1d7b8cdde912" (UID: "f76f8240-ab05-410c-8a60-1d7b8cdde912"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:42:24.291952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.291901    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-token" pod "f76f8240-ab05-410c-8a60-1d7b8cdde912" (UID: "f76f8240-ab05-410c-8a60-1d7b8cdde912"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:42:24.291952 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.291917    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-kube-api-access-wvzpv" pod "f76f8240-ab05-410c-8a60-1d7b8cdde912" (UID: "f76f8240-ab05-410c-8a60-1d7b8cdde912"). InnerVolumeSpecName "kube-api-access-wvzpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:42:24.391006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.390969    5659 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76f8240-ab05-410c-8a60-1d7b8cdde912-local-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:24.391006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.391001    5659 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-token\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:24.391006 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.391011    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvzpv\" (UniqueName: \"kubernetes.io/projected/f76f8240-ab05-410c-8a60-1d7b8cdde912-kube-api-access-wvzpv\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:24.391250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.391021    5659 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-ca-configmap\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:24.391250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.391030    5659 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-csr-dns-cert\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:24.391250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.391038    5659 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-cacerts\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:24.391250 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.391048    5659 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f76f8240-ab05-410c-8a60-1d7b8cdde912-istio-kubeconfig\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:24.757813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.757779    5659 generic.go:356] "Generic (PLEG): container finished" podID="f76f8240-ab05-410c-8a60-1d7b8cdde912" containerID="967f7fd677ea9e5d84ce8df0fd78ecbf8199f16e64bd2c017a91c1ba2a59b49b" exitCode=0
Nov 03 20:42:24.758010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.757830    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"
Nov 03 20:42:24.758010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.757872    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r" event={"ID":"f76f8240-ab05-410c-8a60-1d7b8cdde912","Type":"ContainerDied","Data":"967f7fd677ea9e5d84ce8df0fd78ecbf8199f16e64bd2c017a91c1ba2a59b49b"}
Nov 03 20:42:24.758010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.757930    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r" event={"ID":"f76f8240-ab05-410c-8a60-1d7b8cdde912","Type":"ContainerDied","Data":"3dc1eff39848d7307d10c135b94e8283bc50d6fa9396a60e9dad7eefe689129b"}
Nov 03 20:42:24.758010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.757957    5659 scope.go:122] "RemoveContainer" containerID="967f7fd677ea9e5d84ce8df0fd78ecbf8199f16e64bd2c017a91c1ba2a59b49b"
Nov 03 20:42:24.769156 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.769138    5659 scope.go:122] "RemoveContainer" containerID="967f7fd677ea9e5d84ce8df0fd78ecbf8199f16e64bd2c017a91c1ba2a59b49b"
Nov 03 20:42:24.769415 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:42:24.769387    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967f7fd677ea9e5d84ce8df0fd78ecbf8199f16e64bd2c017a91c1ba2a59b49b\": container with ID starting with 967f7fd677ea9e5d84ce8df0fd78ecbf8199f16e64bd2c017a91c1ba2a59b49b not found: ID does not exist" containerID="967f7fd677ea9e5d84ce8df0fd78ecbf8199f16e64bd2c017a91c1ba2a59b49b"
Nov 03 20:42:24.769545 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.769419    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967f7fd677ea9e5d84ce8df0fd78ecbf8199f16e64bd2c017a91c1ba2a59b49b"} err="failed to get container status \"967f7fd677ea9e5d84ce8df0fd78ecbf8199f16e64bd2c017a91c1ba2a59b49b\": rpc error: code = NotFound desc = could not find container \"967f7fd677ea9e5d84ce8df0fd78ecbf8199f16e64bd2c017a91c1ba2a59b49b\": container with ID starting with 967f7fd677ea9e5d84ce8df0fd78ecbf8199f16e64bd2c017a91c1ba2a59b49b not found: ID does not exist"
Nov 03 20:42:24.781512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.781468    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"]
Nov 03 20:42:24.785066 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:24.785042    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2vf4r"]
Nov 03 20:42:25.967754 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:25.967713    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76f8240-ab05-410c-8a60-1d7b8cdde912" path="/var/lib/kubelet/pods/f76f8240-ab05-410c-8a60-1d7b8cdde912/volumes"
Nov 03 20:42:34.620758 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:34.620726    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:34.751381 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:34.751347    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:45.217360 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:42:45.217321    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:42:47.614813 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:47.614772    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:50.342773 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:42:50.342740    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:00.650746 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:43:00.650707    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:00.797205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:43:00.797169    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:06.680034 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:43:06.679995    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:15.216832 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:43:15.216784    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:43:35.036932 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:43:35.036873    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:35.168186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:43:35.168154    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:45.218610 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:43:45.218519    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:43:45.943159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:43:45.942679    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tnfsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:43:45.943159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:43:45.942748    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tnfsk" podUID="9fa9e0c2-1bfa-4b41-b033-8d286b4c63be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:44:06.748546 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:44:06.748507    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:15.217675 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:44:15.217638    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:44:43.181961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:44:43.181929    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:43.473126 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:44:43.473088    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:45.218474 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:44:45.218437    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:44:47.879507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:44:47.879026    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:06.789201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:45:06.789165    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:15.217015 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:45:15.216935    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:45:35.779598 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:45:35.779562    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:35.913806 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:45:35.913757    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:44.812930 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:45:44.812898    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:44.940065 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:45:44.939993    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:45.217979 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:45:45.217933    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:46:06.859463 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:06.859427    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:09.472680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:09.472637    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:10.300236 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:10.300183    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:10.428050 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:10.428012    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:15.217586 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:46:15.217549    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:46:23.326465 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:23.326428    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:23.470399 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:23.470362    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:24.537459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:24.537425    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:24.686686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:24.686652    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:24.819723 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:24.819626    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:25.333827 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:25.333782    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:26.149507 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:26.149405    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:26.847752 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:26.847705    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:27.526991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:27.526954    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:28.284709 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:28.284665    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:29.007603 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:29.007567    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:29.807872 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:29.807841    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:30.521068 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:30.521033    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:31.218229 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:31.218194    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:32.034552 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:32.034503    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:32.835204 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:32.835143    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:33.187224 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:33.187077    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:33.607094 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:33.607053    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:33.791662 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:33.791626    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:33.927941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:33.927854    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:34.058526 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:34.058463    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:34.490265 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:34.490231    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:35.245113 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:35.245035    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:35.720736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:35.720697    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:36.157228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:36.155271    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:36.916656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:36.916619    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:37.625207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:37.625157    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:38.425685 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:38.425648    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:39.121043 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:39.120955    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:39.823136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:39.823106    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:40.519889 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:40.519852    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:41.220761 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:41.220729    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:41.918011 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:41.917976    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:42.612554 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:42.612510    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:43.312423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:43.312393    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:44.354539 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:44.354500    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:44.726207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:44.726166    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:45.216953 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:46:45.216918    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:46:45.444239 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:45.444199    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:46.225908 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:46.225875    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:47.025877 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:47.025842    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:47.715703 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:47.715664    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:48.499652 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:48.499615    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:49.310898 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:49.310849    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:50.028622 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:50.028553    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:50.731995 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:50.731895    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:51.502344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:51.502309    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:52.137261 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:52.137226    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:52.827131 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:52.827090    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:53.724687 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:53.724655    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:54.530364 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:54.530335    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:55.228259 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:55.228219    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:55.917306 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:55.917270    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:56.618672 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:56.618639    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:57.436640 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:57.436472    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:58.114431 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:58.114391    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:58.827845 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:58.827809    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:59.519207 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:46:59.519171    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:00.224076 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:00.224043    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:00.916240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:00.916200    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:01.402361 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:01.402320    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:01.549061 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:01.549024    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:01.685339 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:01.685260    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:02.414910 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:02.414873    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:03.119359 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:03.119323    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:03.919159 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:03.919122    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:04.612729 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:04.612695    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:05.328651 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:05.328598    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:06.031840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:06.031798    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:06.753864 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:06.753818    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:06.917773 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:06.917735    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:07.167691 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:07.167608    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:07.418253 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:07.418168    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:08.117562 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:08.117516    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:08.834455 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:08.834411    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:09.515774 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:09.515736    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:10.326111 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:10.326076    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:11.026656 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:11.026594    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:11.725408 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:11.725369    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:12.530054 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:12.530015    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:13.220019 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:13.219984    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:13.922861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:13.922822    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:14.617041 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:14.617002    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:15.217138 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:47:15.217103    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:47:15.418644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:15.418591    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:16.123349 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:16.123310    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:16.276086 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:16.276038    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:16.744805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:16.744767    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:16.868644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:16.868608    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:17.631556 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:17.631521    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:18.311361 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:18.311327    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:19.009511 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:19.009470    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:19.720541 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:19.720505    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:20.417644 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:20.417608    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:21.123089 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:21.123055    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:21.438540 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:21.438428    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:21.829263 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:21.829230    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:22.538602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:22.538567    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:23.255154 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:23.255122    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:23.387349 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:23.387317    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:23.632286 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:23.632210    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:23.971584 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:23.964689    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:24.517041 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:24.516943    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:29.617440 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:29.617321    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:29.758284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:29.758246    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:45.216940 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:47:45.216898    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:47:54.717639 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:54.717608    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:54.867602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:54.867560    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:55.891288 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:55.891241    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:56.042652 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:56.042606    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:56.731680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:56.731645    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:57.534533 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:57.534503    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:58.115214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:58.115179    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:58.252079 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:58.252038    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:59.123014 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:59.122982    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:59.331606 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:59.331571    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:59.926650 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:47:59.926618    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:00.750252 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:00.750177    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:01.535366 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:01.535309    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:02.242072 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:02.242034    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:02.917441 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:02.917401    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:03.618185 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:03.618149    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:04.349297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:04.349261    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:05.025123 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:05.025091    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:05.723093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:05.723047    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:05.857728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:05.857693    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:06.421191 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:06.421152    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:06.957138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:06.957108    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:07.143991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:07.141363    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:07.283567 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:07.283447    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:07.813737 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:07.813703    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:08.525284 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:08.525246    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:09.207715 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:09.207679    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:10.011313 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:10.011279    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:10.731292 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:10.731184    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:11.521304 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:11.521261    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:12.220593 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:12.220555    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:12.918468 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:12.918431    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:13.612031 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:13.611989    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:14.419937 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:14.419896    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:14.558205 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:14.558155    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:14.996098 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:14.996035    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:15.154240 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:15.154195    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:15.264084 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:48:15.263937    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:48:15.290780 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:15.290747    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:15.629518 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:15.629405    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:19.574647 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:19.574602    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:24.516750 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:24.516678    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:36.697882 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:48:36.697842    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:45.217242 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:48:45.217199    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:49:07.064678 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:49:07.064638    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:07.349063 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:49:07.348972    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:11.083525 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:49:11.083468    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:11.218276 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:49:11.218240    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:14.420129 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:49:14.420089    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:15.095270 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:49:15.095225    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:15.218710 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:49:15.218669    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:49:15.219840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:49:15.219811    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:15.930854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:49:15.930797    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:18.892523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:49:18.891641    5659 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-g44hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:49:18.892523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:49:18.891716    5659 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-g44hx" podUID="a16a0bd3-5dd9-496b-9708-ee0c4091090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:49:38.225234 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:49:38.225199    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:45.217888 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:49:45.217804    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:50:07.118811 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:50:07.118770    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:07.398692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:50:07.398605    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:08.358684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:50:08.358635    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:08.491701 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:50:08.491667    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:15.216986 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:50:15.216941    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:50:30.855771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:50:30.855740    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:31.021838 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:50:31.021800    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:31.229116 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:50:31.229086    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:45.217249 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:50:45.217203    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:51:07.169082 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:51:07.169043    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:15.218169 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:51:15.218085    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:51:19.245782 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:51:19.245745    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:45.216911 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:51:45.216870    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:51:56.055147 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:51:56.055109    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:07.186496 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:52:07.186457    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:15.216365 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:52:15.216322    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:52:17.702289 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:52:17.702248    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:17.891563 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:52:17.891528    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:18.075219 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:52:18.075175    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:45.216357 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:52:45.216271    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:52:58.209093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:52:58.209059    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:52:59.200971 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:52:59.200937    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:07.276555 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:53:07.276519    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:53:15.217079 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:53:15.217040    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:53:45.217498 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:53:45.217448    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:54:07.358597 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:54:07.358560    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:54:15.222925 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:54:15.222828    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:54:45.216211 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:54:45.216173    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:55:07.397841 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:55:07.397805    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:15.217392 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:55:15.217352    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:55:31.027736 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:55:31.027696    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:31.971840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:55:31.971800    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:55:45.217435 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:55:45.217354    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:56:07.454754 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:56:07.454713    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:15.218513 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:56:15.218447    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:56:39.282269 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:56:39.282226    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:39.405199 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:56:39.405164    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:56:45.216891 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:56:45.216855    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:57:07.504148 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:07.504063    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:15.217314 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:57:15.217227    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:57:16.030078 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:16.030043    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:16.500824 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:16.500778    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:16.634549 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:16.634519    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:19.507764 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:19.507731    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:19.639172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:19.639134    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:20.742297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:20.742254    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:20.880112 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:20.880073    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:21.726686 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:21.726644    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:22.420458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:22.420416    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:23.141990 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:23.141951    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:23.273084 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:23.273038    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:23.940840 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:23.940802    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:24.614413 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:24.614362    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:25.319684 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:25.319643    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:26.018796 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:26.018764    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:26.743728 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:26.737138    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:27.426565 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:27.426525    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:28.137053 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:28.137022    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:28.932771 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:28.932706    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:29.618700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:29.618666    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:30.368370 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:30.368336    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:31.128122 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:31.128087    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:31.826175 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:31.826138    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:32.521126 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:32.521091    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:33.236397 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:33.236364    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:33.930512 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:33.930465    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:34.518423 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:34.518384    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:34.661033 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:34.660999    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:35.121138 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:35.121076    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:35.258897 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:35.258850    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:35.400646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:35.400568    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:35.551178 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:35.551140    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:36.316890 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:36.316852    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:36.546316 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:36.546285    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:37.203961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:37.202062    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:37.939380 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:37.939346    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:38.812613 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:38.812575    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:39.634448 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:39.634412    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:40.343435 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:40.343405    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:41.033007 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:41.032971    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:41.743938 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:41.743905    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:42.531172 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:42.531138    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:43.219710 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:43.219657    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:43.930180 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:43.926626    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:44.622382 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:44.622334    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:45.218189 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:57:45.218151    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:57:45.328615 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:45.328549    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:46.026638 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:46.026605    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:46.726777 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:46.726740    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:47.526851 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:47.526808    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:48.218459 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:48.218425    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:49.019194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:49.019156    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:49.729523 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:49.728651    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:50.437297 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:50.437234    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:51.136214 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:51.136156    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:51.824619 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:51.824582    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:52.723973 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:52.723936    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:53.408646 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:53.408610    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:54.215473 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:54.215423    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:55.114581 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:55.114545    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:55.870874 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:55.870840    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:56.626972 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:56.626939    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:57.326503 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:57.326461    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:58.016179 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:58.016141    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:58.723223 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:58.723192    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:57:59.418991 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:57:59.418957    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:00.146099 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:00.146065    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:00.821756 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:00.821722    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:01.529961 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:01.529926    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:02.240979 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:02.240039    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:02.922807 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:02.922767    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:03.619711 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:03.619677    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:04.352842 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:04.352804    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:05.027153 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:05.027119    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:05.829274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:05.829237    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:06.530093 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:06.530032    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:07.222009 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:07.221978    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:07.551692 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:07.551611    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:07.920854 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:07.920778    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:08.625218 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:08.625178    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:09.321642 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:09.321598    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:10.029941 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:10.029885    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:10.716849 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:10.716793    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:11.413714 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:11.413681    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:12.225573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:12.225540    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:12.939298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:12.939257    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:13.629246 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:13.629212    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:14.319160 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:14.319126    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:15.021544 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:15.021510    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:15.217701 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:58:15.217666    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:58:15.739634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:15.739599    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:16.422000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:16.421961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:17.125935 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:17.125898    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:17.825915 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:17.825874    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:18.616665 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:18.616627    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:19.334768 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:19.334706    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:20.020208 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:20.020175    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:20.717071 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:20.717039    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:21.414433 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:21.414375    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:22.214858 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:22.214812    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:22.918989 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:22.918951    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:23.049295 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:23.049249    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:23.417909 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:23.417873    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:23.609010 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:23.608971    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:24.321593 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:24.321557    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:25.027042 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:25.027001    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:25.819017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:25.818981    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:26.507308 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:26.507274    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:27.140092 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:27.140060    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:27.814370 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:27.814335    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:28.026392 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:28.026355    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:28.625268 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:28.625229    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:29.322242 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:29.322204    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:29.893140 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:29.893100    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:30.031572 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:30.031535    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:30.163543 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:30.163443    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:30.711274 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:30.711234    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:35.980522 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:35.980478    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:36.117590 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:58:36.117553    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:58:45.216835 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:58:45.216746    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:59:00.992530 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:00.992491    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:01.132587 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:01.132541    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:02.123458 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:02.123414    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:02.272017 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:02.271982    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:02.943649 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:02.943609    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:03.726573 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:03.726536    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:04.232201 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:04.232166    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:04.541365 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:04.541248    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:05.335228 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:05.335187    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:05.955064 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:05.954972    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:06.090602 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:06.089684    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:06.920659 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:06.920618    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:07.603895 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:07.603851    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:07.841492 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:07.841443    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:08.481904 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:08.481856    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:09.137805 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:09.137746    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:09.836884 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:09.836830    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:10.560502 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:10.560453    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:11.217802 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:11.217738    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:11.921000 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:11.920942    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:12.135681 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:12.135644    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:12.272641 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:12.272604    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:12.633309 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:12.633220    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:13.320974 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:13.320943    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:14.017027 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:14.016993    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:14.810631 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:14.810587    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:15.217799 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:59:15.217762    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 20:59:15.511136 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:15.511052    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:16.218994 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:16.218965    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:16.926700 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:16.926668    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:17.641194 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:17.641157    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:18.328969 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:18.328936    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:19.131344 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:19.131308    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:19.822329 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:19.822298    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:20.527586 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:20.527553    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:20.658142 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:20.658104    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:21.161596 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:21.161557    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:21.322621 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:21.322584    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:21.451186 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:21.451102    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:21.751311 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:21.751276    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 20:59:38.445634 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:38.445355    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"]
Nov 03 20:59:38.446679 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:38.445753    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr" podUID="06729d18-3d24-489f-9e57-4b24a6ca6323" containerName="manager" containerID="cri-o://a64053620fcb9a7cdc9f716ac7da3237387f0ec02695f6f8018ebae26d129ec2" gracePeriod=10
Nov 03 20:59:38.730058 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:38.730030    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"
Nov 03 20:59:38.813351 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:38.813313    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06729d18-3d24-489f-9e57-4b24a6ca6323-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06729d18-3d24-489f-9e57-4b24a6ca6323-extensions-socket-volume\") pod \"06729d18-3d24-489f-9e57-4b24a6ca6323\" (UID: \"06729d18-3d24-489f-9e57-4b24a6ca6323\") "
Nov 03 20:59:38.813553 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:38.813369    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06729d18-3d24-489f-9e57-4b24a6ca6323-kube-api-access-5qd55\" (UniqueName: \"kubernetes.io/projected/06729d18-3d24-489f-9e57-4b24a6ca6323-kube-api-access-5qd55\") pod \"06729d18-3d24-489f-9e57-4b24a6ca6323\" (UID: \"06729d18-3d24-489f-9e57-4b24a6ca6323\") "
Nov 03 20:59:38.813732 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:38.813700    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06729d18-3d24-489f-9e57-4b24a6ca6323-extensions-socket-volume" pod "06729d18-3d24-489f-9e57-4b24a6ca6323" (UID: "06729d18-3d24-489f-9e57-4b24a6ca6323"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:38.813878 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:38.813859    5659 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06729d18-3d24-489f-9e57-4b24a6ca6323-extensions-socket-volume\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:38.815680 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:38.815649    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06729d18-3d24-489f-9e57-4b24a6ca6323-kube-api-access-5qd55" pod "06729d18-3d24-489f-9e57-4b24a6ca6323" (UID: "06729d18-3d24-489f-9e57-4b24a6ca6323"). InnerVolumeSpecName "kube-api-access-5qd55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:38.914900 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:38.914849    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qd55\" (UniqueName: \"kubernetes.io/projected/06729d18-3d24-489f-9e57-4b24a6ca6323-kube-api-access-5qd55\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:39.282415 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:39.282378    5659 generic.go:356] "Generic (PLEG): container finished" podID="06729d18-3d24-489f-9e57-4b24a6ca6323" containerID="a64053620fcb9a7cdc9f716ac7da3237387f0ec02695f6f8018ebae26d129ec2" exitCode=0
Nov 03 20:59:39.282627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:39.282425    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"
Nov 03 20:59:39.282627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:39.282460    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr" event={"ID":"06729d18-3d24-489f-9e57-4b24a6ca6323","Type":"ContainerDied","Data":"a64053620fcb9a7cdc9f716ac7da3237387f0ec02695f6f8018ebae26d129ec2"}
Nov 03 20:59:39.282627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:39.282527    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr" event={"ID":"06729d18-3d24-489f-9e57-4b24a6ca6323","Type":"ContainerDied","Data":"de56cb3cfa7a885b5d96d0a56a3f06975239318ee32b4e1ca35d20c6e6d829cb"}
Nov 03 20:59:39.282627 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:39.282550    5659 scope.go:122] "RemoveContainer" containerID="a64053620fcb9a7cdc9f716ac7da3237387f0ec02695f6f8018ebae26d129ec2"
Nov 03 20:59:39.295570 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:39.295549    5659 scope.go:122] "RemoveContainer" containerID="a64053620fcb9a7cdc9f716ac7da3237387f0ec02695f6f8018ebae26d129ec2"
Nov 03 20:59:39.295805 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:59:39.295785    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a64053620fcb9a7cdc9f716ac7da3237387f0ec02695f6f8018ebae26d129ec2\": container with ID starting with a64053620fcb9a7cdc9f716ac7da3237387f0ec02695f6f8018ebae26d129ec2 not found: ID does not exist" containerID="a64053620fcb9a7cdc9f716ac7da3237387f0ec02695f6f8018ebae26d129ec2"
Nov 03 20:59:39.295861 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:39.295815    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a64053620fcb9a7cdc9f716ac7da3237387f0ec02695f6f8018ebae26d129ec2"} err="failed to get container status \"a64053620fcb9a7cdc9f716ac7da3237387f0ec02695f6f8018ebae26d129ec2\": rpc error: code = NotFound desc = could not find container \"a64053620fcb9a7cdc9f716ac7da3237387f0ec02695f6f8018ebae26d129ec2\": container with ID starting with a64053620fcb9a7cdc9f716ac7da3237387f0ec02695f6f8018ebae26d129ec2 not found: ID does not exist"
Nov 03 20:59:39.306131 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:39.306106    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"]
Nov 03 20:59:39.312110 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:39.312087    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-l8fcr"]
Nov 03 20:59:39.968298 ip-10-0-133-249 kubenswrapper[5659]: I1103 20:59:39.968260    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06729d18-3d24-489f-9e57-4b24a6ca6323" path="/var/lib/kubelet/pods/06729d18-3d24-489f-9e57-4b24a6ca6323/volumes"
Nov 03 20:59:45.216425 ip-10-0-133-249 kubenswrapper[5659]: E1103 20:59:45.216385    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:00:07.669876 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:00:07.669844    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:15.216799 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:00:15.216709    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:00:30.471397 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:00:30.471363    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:00:45.216744 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:00:45.216702    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:01:07.722398 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:07.722359    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:13.918709 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:13.918670    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:01:15.216796 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:01:15.216753    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:01:45.216645 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:01:45.216562    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:01:50.953837 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:50.953804    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"]
Nov 03 21:01:50.954475 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:50.954049    5659 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2" podUID="6ffb51a7-07f2-4ae8-91da-250f92d7c155" containerName="maas-api" containerID="cri-o://be081e1afbe0e73dc718dd0794486a41ce4c11dea535612addeac2b8e8893672" gracePeriod=30
Nov 03 21:01:52.211335 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:52.211306    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 21:01:52.284778 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:52.284738    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ffb51a7-07f2-4ae8-91da-250f92d7c155-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ffb51a7-07f2-4ae8-91da-250f92d7c155-openshift-service-ca\") pod \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") "
Nov 03 21:01:52.284976 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:52.284807    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-metrics-certs\") pod \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") "
Nov 03 21:01:52.284976 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:52.284871    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ffb51a7-07f2-4ae8-91da-250f92d7c155-kube-api-access-hr27q\" (UniqueName: \"kubernetes.io/projected/6ffb51a7-07f2-4ae8-91da-250f92d7c155-kube-api-access-hr27q\") pod \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") "
Nov 03 21:01:52.284976 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:52.284955    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-maas-api-tls\") pod \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\" (UID: \"6ffb51a7-07f2-4ae8-91da-250f92d7c155\") "
Nov 03 21:01:52.285139 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:52.285085    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffb51a7-07f2-4ae8-91da-250f92d7c155-openshift-service-ca" pod "6ffb51a7-07f2-4ae8-91da-250f92d7c155" (UID: "6ffb51a7-07f2-4ae8-91da-250f92d7c155"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:52.285320 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:52.285298    5659 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ffb51a7-07f2-4ae8-91da-250f92d7c155-openshift-service-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:52.287093 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:52.287067    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-metrics-certs" pod "6ffb51a7-07f2-4ae8-91da-250f92d7c155" (UID: "6ffb51a7-07f2-4ae8-91da-250f92d7c155"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:01:52.287182 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:52.287140    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-maas-api-tls" pod "6ffb51a7-07f2-4ae8-91da-250f92d7c155" (UID: "6ffb51a7-07f2-4ae8-91da-250f92d7c155"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:01:52.287226 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:52.287195    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffb51a7-07f2-4ae8-91da-250f92d7c155-kube-api-access-hr27q" pod "6ffb51a7-07f2-4ae8-91da-250f92d7c155" (UID: "6ffb51a7-07f2-4ae8-91da-250f92d7c155"). InnerVolumeSpecName "kube-api-access-hr27q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:52.386822 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:52.386779    5659 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-metrics-certs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:52.386822 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:52.386816    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr27q\" (UniqueName: \"kubernetes.io/projected/6ffb51a7-07f2-4ae8-91da-250f92d7c155-kube-api-access-hr27q\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:52.387045 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:52.386833    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ffb51a7-07f2-4ae8-91da-250f92d7c155-maas-api-tls\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:53.011893 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:53.011852    5659 generic.go:356] "Generic (PLEG): container finished" podID="6ffb51a7-07f2-4ae8-91da-250f92d7c155" containerID="be081e1afbe0e73dc718dd0794486a41ce4c11dea535612addeac2b8e8893672" exitCode=0
Nov 03 21:01:53.012080 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:53.011943    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"
Nov 03 21:01:53.012080 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:53.011940    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2" event={"ID":"6ffb51a7-07f2-4ae8-91da-250f92d7c155","Type":"ContainerDied","Data":"be081e1afbe0e73dc718dd0794486a41ce4c11dea535612addeac2b8e8893672"}
Nov 03 21:01:53.012173 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:53.012096    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2" event={"ID":"6ffb51a7-07f2-4ae8-91da-250f92d7c155","Type":"ContainerDied","Data":"4ec07f2d9d2ed6ecf8cef9eb2285c85e5b0e08f28f2b1421f8b2ef10a385caaf"}
Nov 03 21:01:53.012173 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:53.012120    5659 scope.go:122] "RemoveContainer" containerID="be081e1afbe0e73dc718dd0794486a41ce4c11dea535612addeac2b8e8893672"
Nov 03 21:01:53.023218 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:53.023194    5659 scope.go:122] "RemoveContainer" containerID="be081e1afbe0e73dc718dd0794486a41ce4c11dea535612addeac2b8e8893672"
Nov 03 21:01:53.023510 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:01:53.023473    5659 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be081e1afbe0e73dc718dd0794486a41ce4c11dea535612addeac2b8e8893672\": container with ID starting with be081e1afbe0e73dc718dd0794486a41ce4c11dea535612addeac2b8e8893672 not found: ID does not exist" containerID="be081e1afbe0e73dc718dd0794486a41ce4c11dea535612addeac2b8e8893672"
Nov 03 21:01:53.023581 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:53.023521    5659 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be081e1afbe0e73dc718dd0794486a41ce4c11dea535612addeac2b8e8893672"} err="failed to get container status \"be081e1afbe0e73dc718dd0794486a41ce4c11dea535612addeac2b8e8893672\": rpc error: code = NotFound desc = could not find container \"be081e1afbe0e73dc718dd0794486a41ce4c11dea535612addeac2b8e8893672\": container with ID starting with be081e1afbe0e73dc718dd0794486a41ce4c11dea535612addeac2b8e8893672 not found: ID does not exist"
Nov 03 21:01:53.035771 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:53.035731    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"]
Nov 03 21:01:53.037583 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:53.037558    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cd7866f98-ntwg2"]
Nov 03 21:01:53.968735 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:01:53.968690    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ffb51a7-07f2-4ae8-91da-250f92d7c155" path="/var/lib/kubelet/pods/6ffb51a7-07f2-4ae8-91da-250f92d7c155/volumes"
Nov 03 21:02:03.482508 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482458    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"]
Nov 03 21:02:03.482918 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482894    5659 cpu_manager.go:414] "containerMap: removing container" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy"
Nov 03 21:02:03.482918 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482905    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy"
Nov 03 21:02:03.482918 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482918    5659 cpu_manager.go:414] "containerMap: removing container" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482923    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482935    5659 cpu_manager.go:414] "containerMap: removing container" podUID="06729d18-3d24-489f-9e57-4b24a6ca6323" containerName="manager"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482940    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06729d18-3d24-489f-9e57-4b24a6ca6323" containerName="manager"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482950    5659 cpu_manager.go:414] "containerMap: removing container" podUID="cce0df39-f9a8-4347-aa52-423d78661bfe" containerName="payload-pre-processing"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482955    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce0df39-f9a8-4347-aa52-423d78661bfe" containerName="payload-pre-processing"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482963    5659 cpu_manager.go:414] "containerMap: removing container" podUID="aba237d4-70b3-4347-86b6-462fc504fe3d" containerName="istio-proxy"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482968    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba237d4-70b3-4347-86b6-462fc504fe3d" containerName="istio-proxy"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482976    5659 cpu_manager.go:414] "containerMap: removing container" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482980    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482988    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerName="istio-proxy"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482992    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerName="istio-proxy"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.482999    5659 cpu_manager.go:414] "containerMap: removing container" podUID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerName="istio-proxy"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483003    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerName="istio-proxy"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483009    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3079f0fd-1225-42ab-8151-4062129bcca0" containerName="payload-processing"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483013    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3079f0fd-1225-42ab-8151-4062129bcca0" containerName="payload-processing"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483021    5659 cpu_manager.go:414] "containerMap: removing container" podUID="376c36bb-8e35-46d3-9ced-b2a66f7370f7" containerName="payload-processing"
Nov 03 21:02:03.483023 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483026    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376c36bb-8e35-46d3-9ced-b2a66f7370f7" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483035    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bee89175-06af-4e58-8b04-bd892b530f1e" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483040    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee89175-06af-4e58-8b04-bd892b530f1e" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483046    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c397c76f-9780-496f-bb87-372a06c94d40" containerName="payload-pre-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483051    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c397c76f-9780-496f-bb87-372a06c94d40" containerName="payload-pre-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483060    5659 cpu_manager.go:414] "containerMap: removing container" podUID="fdde0ea7-99e2-4098-8e03-db9b8533aab9" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483065    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdde0ea7-99e2-4098-8e03-db9b8533aab9" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483073    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b793166f-3de9-4167-97d6-e3cd33c7a72f" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483078    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b793166f-3de9-4167-97d6-e3cd33c7a72f" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483084    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483088    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483097    5659 cpu_manager.go:414] "containerMap: removing container" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483101    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483111    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6c85f57d-183f-408c-a450-6d6ad119a7db" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483115    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c85f57d-183f-408c-a450-6d6ad119a7db" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483122    5659 cpu_manager.go:414] "containerMap: removing container" podUID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483127    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483134    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483138    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483145    5659 cpu_manager.go:414] "containerMap: removing container" podUID="7c195f09-b9ef-45a0-89d6-07ea2a203277" containerName="payload-pre-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483150    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c195f09-b9ef-45a0-89d6-07ea2a203277" containerName="payload-pre-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483156    5659 cpu_manager.go:414] "containerMap: removing container" podUID="fbf0a278-c34b-484d-9b5e-11243d3c2c63" containerName="payload-pre-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483160    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf0a278-c34b-484d-9b5e-11243d3c2c63" containerName="payload-pre-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483166    5659 cpu_manager.go:414] "containerMap: removing container" podUID="b039d7a8-14c3-4195-9967-121992120988" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483170    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b039d7a8-14c3-4195-9967-121992120988" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483176    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bd928a60-b898-49f3-8ff0-5dfc52ef4fe4" containerName="maas-api"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483180    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd928a60-b898-49f3-8ff0-5dfc52ef4fe4" containerName="maas-api"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483191    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f76f8240-ab05-410c-8a60-1d7b8cdde912" containerName="discovery"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483195    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76f8240-ab05-410c-8a60-1d7b8cdde912" containerName="discovery"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483202    5659 cpu_manager.go:414] "containerMap: removing container" podUID="be08436f-abba-4a9d-9aa3-1f9aa05d883e" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483206    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be08436f-abba-4a9d-9aa3-1f9aa05d883e" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483212    5659 cpu_manager.go:414] "containerMap: removing container" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483216    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483226    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d468b03c-81e8-44a3-948e-ed1e86de9aca" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483230    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d468b03c-81e8-44a3-948e-ed1e86de9aca" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483239    5659 cpu_manager.go:414] "containerMap: removing container" podUID="57bc1738-e920-4570-b31b-9e4937df03b0" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483243    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57bc1738-e920-4570-b31b-9e4937df03b0" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483249    5659 cpu_manager.go:414] "containerMap: removing container" podUID="a7356100-2a57-4289-b38f-133d56fa44d1" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483254    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7356100-2a57-4289-b38f-133d56fa44d1" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483261    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d6fff012-3266-4752-b836-fe60b7e4fb2b" containerName="maas-api"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483266    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6fff012-3266-4752-b836-fe60b7e4fb2b" containerName="maas-api"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483273    5659 cpu_manager.go:414] "containerMap: removing container" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483286    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483293    5659 cpu_manager.go:414] "containerMap: removing container" podUID="821779e9-42f1-4acc-8771-39d4e55f9abf" containerName="maas-api"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483298    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821779e9-42f1-4acc-8771-39d4e55f9abf" containerName="maas-api"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483305    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c981a786-15c1-4282-9823-8a937781f2fe" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483310    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c981a786-15c1-4282-9823-8a937781f2fe" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483317    5659 cpu_manager.go:414] "containerMap: removing container" podUID="d4922de3-227d-479c-a91d-1db7e320b89b" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483321    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4922de3-227d-479c-a91d-1db7e320b89b" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483327    5659 cpu_manager.go:414] "containerMap: removing container" podUID="992549e4-0f0c-4a0e-a170-a472285aa24f" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483333    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992549e4-0f0c-4a0e-a170-a472285aa24f" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483342    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bfcbe9c0-443c-4551-93f9-1b50010ff9f0" containerName="payload-pre-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483346    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfcbe9c0-443c-4551-93f9-1b50010ff9f0" containerName="payload-pre-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483352    5659 cpu_manager.go:414] "containerMap: removing container" podUID="6ffb51a7-07f2-4ae8-91da-250f92d7c155" containerName="maas-api"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483357    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffb51a7-07f2-4ae8-91da-250f92d7c155" containerName="maas-api"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483418    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="06729d18-3d24-489f-9e57-4b24a6ca6323" containerName="manager"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483425    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="0168a66b-a1c3-4bf2-a921-8cdff64b3ed4" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483430    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce0df39-f9a8-4347-aa52-423d78661bfe" containerName="payload-pre-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483438    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba237d4-70b3-4347-86b6-462fc504fe3d" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483446    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fcdaebb-cfd8-451c-aae7-fd3603ffdfa9" containerName="main"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483453    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483459    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="8596f4f0-3ef7-446c-8374-57bcf455301c" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483465    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3079f0fd-1225-42ab-8151-4062129bcca0" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483470    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="376c36bb-8e35-46d3-9ced-b2a66f7370f7" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483475    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee89175-06af-4e58-8b04-bd892b530f1e" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483493    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c397c76f-9780-496f-bb87-372a06c94d40" containerName="payload-pre-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483502    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdde0ea7-99e2-4098-8e03-db9b8533aab9" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483510    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b793166f-3de9-4167-97d6-e3cd33c7a72f" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483520    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d9f7b7-51af-4be2-b959-3143ebc16f58" containerName="main"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483528    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c95cd0-cf66-4a36-9c3e-f917ee445790" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483536    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c85f57d-183f-408c-a450-6d6ad119a7db" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483546    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c08e50-44dd-4bf9-ae14-f24e2672e4c5" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483554    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1cb0655-e9f8-4d24-b524-5adbe643ba9d" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483564    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c195f09-b9ef-45a0-89d6-07ea2a203277" containerName="payload-pre-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483572    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf0a278-c34b-484d-9b5e-11243d3c2c63" containerName="payload-pre-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483579    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="b039d7a8-14c3-4195-9967-121992120988" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483584    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd928a60-b898-49f3-8ff0-5dfc52ef4fe4" containerName="maas-api"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483589    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76f8240-ab05-410c-8a60-1d7b8cdde912" containerName="discovery"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483594    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="be08436f-abba-4a9d-9aa3-1f9aa05d883e" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483598    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="72080325-72f5-4713-a185-570a01746fc8" containerName="main"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483605    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d468b03c-81e8-44a3-948e-ed1e86de9aca" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483612    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="57bc1738-e920-4570-b31b-9e4937df03b0" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483617    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7356100-2a57-4289-b38f-133d56fa44d1" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483623    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6fff012-3266-4752-b836-fe60b7e4fb2b" containerName="maas-api"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483628    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="11abc9df-4f9d-4e6b-b1cf-5f2e64db0b4a" containerName="main"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483634    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="821779e9-42f1-4acc-8771-39d4e55f9abf" containerName="maas-api"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483638    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c981a786-15c1-4282-9823-8a937781f2fe" containerName="payload-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483643    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4922de3-227d-479c-a91d-1db7e320b89b" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483649    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="992549e4-0f0c-4a0e-a170-a472285aa24f" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483655    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfcbe9c0-443c-4551-93f9-1b50010ff9f0" containerName="payload-pre-processing"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483662    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffb51a7-07f2-4ae8-91da-250f92d7c155" containerName="maas-api"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483668    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdffad9b-17e0-408b-b096-73ca58b439b3" containerName="istio-proxy"
Nov 03 21:02:03.483650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.483675    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ffe4fd-3411-41b3-898f-44da3191ee57" containerName="main"
Nov 03 21:02:03.488461 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.488444    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"
Nov 03 21:02:03.493388 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.493367    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:02:03.493559 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.493391    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:02:03.493559 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.493408    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:02:03.493559 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.493457    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:02:03.497007 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.496708    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"]
Nov 03 21:02:03.512090 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.512052    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a029e6c-qdmz4\" (UID: \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"
Nov 03 21:02:03.512520 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.512452    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzzbz\" (UniqueName: \"kubernetes.io/projected/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-kube-api-access-tzzbz\") pod \"maas-api-revoke-sub-9a029e6c-qdmz4\" (UID: \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"
Nov 03 21:02:03.512653 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.512526    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a029e6c-qdmz4\" (UID: \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"
Nov 03 21:02:03.613265 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.613224    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a029e6c-qdmz4\" (UID: \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"
Nov 03 21:02:03.613454 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.613326    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzzbz\" (UniqueName: \"kubernetes.io/projected/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-kube-api-access-tzzbz\") pod \"maas-api-revoke-sub-9a029e6c-qdmz4\" (UID: \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"
Nov 03 21:02:03.613454 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.613352    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a029e6c-qdmz4\" (UID: \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"
Nov 03 21:02:03.613986 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.613955    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a029e6c-qdmz4\" (UID: \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"
Nov 03 21:02:03.615742 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.615722    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a029e6c-qdmz4\" (UID: \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"
Nov 03 21:02:03.625693 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.625667    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzzbz\" (UniqueName: \"kubernetes.io/projected/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-kube-api-access-tzzbz\") pod \"maas-api-revoke-sub-9a029e6c-qdmz4\" (UID: \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"
Nov 03 21:02:03.804560 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:03.804460    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"
Nov 03 21:02:04.080627 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:04.080596    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"]
Nov 03 21:02:04.082746 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:02:04.082717    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121b8fc8_36c2_4e69_ac1e_1fffe3d0e72a.slice/crio-c0f1ca4e64a3fe53849f098fe101977f66172a636a4b39c7bbb66d4a666b822b WatchSource:0}: Error finding container c0f1ca4e64a3fe53849f098fe101977f66172a636a4b39c7bbb66d4a666b822b: Status 404 returned error can't find the container with id c0f1ca4e64a3fe53849f098fe101977f66172a636a4b39c7bbb66d4a666b822b
Nov 03 21:02:05.084635 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:05.084595    5659 generic.go:356] "Generic (PLEG): container finished" podID="121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a" containerID="aba3a70937ea2ea597ea0db77c1bfb1decf2a2ab64134f58e7373eee994af042" exitCode=0
Nov 03 21:02:05.085048 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:05.084681    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4" event={"ID":"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a","Type":"ContainerDied","Data":"aba3a70937ea2ea597ea0db77c1bfb1decf2a2ab64134f58e7373eee994af042"}
Nov 03 21:02:05.085048 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:05.084731    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4" event={"ID":"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a","Type":"ContainerStarted","Data":"c0f1ca4e64a3fe53849f098fe101977f66172a636a4b39c7bbb66d4a666b822b"}
Nov 03 21:02:06.234432 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:06.234404    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"
Nov 03 21:02:06.339222 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:06.339191    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-xks-ca\") pod \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\" (UID: \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\") "
Nov 03 21:02:06.339441 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:06.339318    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-openshift-ca\") pod \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\" (UID: \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\") "
Nov 03 21:02:06.339441 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:06.339395    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-kube-api-access-tzzbz\" (UniqueName: \"kubernetes.io/projected/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-kube-api-access-tzzbz\") pod \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\" (UID: \"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a\") "
Nov 03 21:02:06.339702 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:06.339678    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-openshift-ca" pod "121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a" (UID: "121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:02:06.341314 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:06.341258    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-xks-ca" pod "121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a" (UID: "121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:02:06.341422 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:06.341335    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-kube-api-access-tzzbz" pod "121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a" (UID: "121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a"). InnerVolumeSpecName "kube-api-access-tzzbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:06.440463 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:06.440433    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:06.440463 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:06.440462    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzzbz\" (UniqueName: \"kubernetes.io/projected/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-kube-api-access-tzzbz\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:06.440663 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:06.440473    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:07.096306 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:07.096273    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"
Nov 03 21:02:07.096306 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:07.096281    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4" event={"ID":"121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a","Type":"ContainerDied","Data":"c0f1ca4e64a3fe53849f098fe101977f66172a636a4b39c7bbb66d4a666b822b"}
Nov 03 21:02:07.096532 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:07.096330    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0f1ca4e64a3fe53849f098fe101977f66172a636a4b39c7bbb66d4a666b822b"
Nov 03 21:02:07.781529 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:07.781499    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:02:15.216001 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:02:15.215962    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:02:23.686178 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.686134    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"]
Nov 03 21:02:23.686726 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.686614    5659 cpu_manager.go:414] "containerMap: removing container" podUID="121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a" containerName="revoke-keys"
Nov 03 21:02:23.686726 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.686627    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a" containerName="revoke-keys"
Nov 03 21:02:23.686726 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.686720    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a" containerName="revoke-keys"
Nov 03 21:02:23.690062 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.690036    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"
Nov 03 21:02:23.694256 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.694236    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:02:23.695549 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.695522    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:02:23.695710 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.695529    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:02:23.695710 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.695534    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:02:23.698858 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.698833    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"]
Nov 03 21:02:23.811079 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.811041    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-901ba1c0-dptzp\" (UID: \"20531b22-e37e-437f-b65f-3afc0560b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"
Nov 03 21:02:23.811271 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.811185    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-901ba1c0-dptzp\" (UID: \"20531b22-e37e-437f-b65f-3afc0560b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"
Nov 03 21:02:23.811271 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.811263    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87zf\" (UniqueName: \"kubernetes.io/projected/20531b22-e37e-437f-b65f-3afc0560b35b-kube-api-access-z87zf\") pod \"maas-api-revoke-sub-901ba1c0-dptzp\" (UID: \"20531b22-e37e-437f-b65f-3afc0560b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"
Nov 03 21:02:23.912610 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.912576    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87zf\" (UniqueName: \"kubernetes.io/projected/20531b22-e37e-437f-b65f-3afc0560b35b-kube-api-access-z87zf\") pod \"maas-api-revoke-sub-901ba1c0-dptzp\" (UID: \"20531b22-e37e-437f-b65f-3afc0560b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"
Nov 03 21:02:23.912783 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.912632    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-901ba1c0-dptzp\" (UID: \"20531b22-e37e-437f-b65f-3afc0560b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"
Nov 03 21:02:23.912783 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.912768    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-901ba1c0-dptzp\" (UID: \"20531b22-e37e-437f-b65f-3afc0560b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"
Nov 03 21:02:23.913329 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.913306    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-901ba1c0-dptzp\" (UID: \"20531b22-e37e-437f-b65f-3afc0560b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"
Nov 03 21:02:23.915092 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.915072    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-901ba1c0-dptzp\" (UID: \"20531b22-e37e-437f-b65f-3afc0560b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"
Nov 03 21:02:23.925100 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:23.925072    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87zf\" (UniqueName: \"kubernetes.io/projected/20531b22-e37e-437f-b65f-3afc0560b35b-kube-api-access-z87zf\") pod \"maas-api-revoke-sub-901ba1c0-dptzp\" (UID: \"20531b22-e37e-437f-b65f-3afc0560b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"
Nov 03 21:02:24.010957 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:24.010924    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"
Nov 03 21:02:24.288213 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:24.288186    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"]
Nov 03 21:02:24.290102 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:02:24.290075    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20531b22_e37e_437f_b65f_3afc0560b35b.slice/crio-91cc59f40d9613484bbb946ae0dd031d9a9291f05f7ef3a690ec8dd0d24f6db7 WatchSource:0}: Error finding container 91cc59f40d9613484bbb946ae0dd031d9a9291f05f7ef3a690ec8dd0d24f6db7: Status 404 returned error can't find the container with id 91cc59f40d9613484bbb946ae0dd031d9a9291f05f7ef3a690ec8dd0d24f6db7
Nov 03 21:02:25.205821 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:25.205786    5659 generic.go:356] "Generic (PLEG): container finished" podID="20531b22-e37e-437f-b65f-3afc0560b35b" containerID="4b53e2957208cff87e0f65a2c07f9922eac3a12b56c53ba677d5638b3c74f834" exitCode=0
Nov 03 21:02:25.206221 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:25.205868    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp" event={"ID":"20531b22-e37e-437f-b65f-3afc0560b35b","Type":"ContainerDied","Data":"4b53e2957208cff87e0f65a2c07f9922eac3a12b56c53ba677d5638b3c74f834"}
Nov 03 21:02:25.206221 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:25.205917    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp" event={"ID":"20531b22-e37e-437f-b65f-3afc0560b35b","Type":"ContainerStarted","Data":"91cc59f40d9613484bbb946ae0dd031d9a9291f05f7ef3a690ec8dd0d24f6db7"}
Nov 03 21:02:26.340390 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:26.340359    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"
Nov 03 21:02:26.440135 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:26.440102    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-openshift-ca\") pod \"20531b22-e37e-437f-b65f-3afc0560b35b\" (UID: \"20531b22-e37e-437f-b65f-3afc0560b35b\") "
Nov 03 21:02:26.440319 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:26.440159    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20531b22-e37e-437f-b65f-3afc0560b35b-kube-api-access-z87zf\" (UniqueName: \"kubernetes.io/projected/20531b22-e37e-437f-b65f-3afc0560b35b-kube-api-access-z87zf\") pod \"20531b22-e37e-437f-b65f-3afc0560b35b\" (UID: \"20531b22-e37e-437f-b65f-3afc0560b35b\") "
Nov 03 21:02:26.440319 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:26.440262    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-xks-ca\") pod \"20531b22-e37e-437f-b65f-3afc0560b35b\" (UID: \"20531b22-e37e-437f-b65f-3afc0560b35b\") "
Nov 03 21:02:26.440465 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:26.440440    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-openshift-ca" pod "20531b22-e37e-437f-b65f-3afc0560b35b" (UID: "20531b22-e37e-437f-b65f-3afc0560b35b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:02:26.440668 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:26.440644    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:26.442323 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:26.442301    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-xks-ca" pod "20531b22-e37e-437f-b65f-3afc0560b35b" (UID: "20531b22-e37e-437f-b65f-3afc0560b35b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:02:26.442410 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:26.442389    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20531b22-e37e-437f-b65f-3afc0560b35b-kube-api-access-z87zf" pod "20531b22-e37e-437f-b65f-3afc0560b35b" (UID: "20531b22-e37e-437f-b65f-3afc0560b35b"). InnerVolumeSpecName "kube-api-access-z87zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:26.542091 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:26.542013    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z87zf\" (UniqueName: \"kubernetes.io/projected/20531b22-e37e-437f-b65f-3afc0560b35b-kube-api-access-z87zf\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:26.542091 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:26.542043    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20531b22-e37e-437f-b65f-3afc0560b35b-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:27.218199 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:27.218165    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"
Nov 03 21:02:27.218381 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:27.218165    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp" event={"ID":"20531b22-e37e-437f-b65f-3afc0560b35b","Type":"ContainerDied","Data":"91cc59f40d9613484bbb946ae0dd031d9a9291f05f7ef3a690ec8dd0d24f6db7"}
Nov 03 21:02:27.218381 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:27.218296    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91cc59f40d9613484bbb946ae0dd031d9a9291f05f7ef3a690ec8dd0d24f6db7"
Nov 03 21:02:43.523707 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.523668    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"]
Nov 03 21:02:43.524320 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.524083    5659 cpu_manager.go:414] "containerMap: removing container" podUID="20531b22-e37e-437f-b65f-3afc0560b35b" containerName="revoke-keys"
Nov 03 21:02:43.524320 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.524094    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20531b22-e37e-437f-b65f-3afc0560b35b" containerName="revoke-keys"
Nov 03 21:02:43.524320 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.524187    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="20531b22-e37e-437f-b65f-3afc0560b35b" containerName="revoke-keys"
Nov 03 21:02:43.527600 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.527576    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"
Nov 03 21:02:43.532295 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.532225    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:02:43.532295 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.532247    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:02:43.533338 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.533300    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:02:43.533464 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.533357    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:02:43.536387 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.536364    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"]
Nov 03 21:02:43.622656 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.622619    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89487c84-fsl55\" (UID: \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"
Nov 03 21:02:43.622827 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.622665    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89487c84-fsl55\" (UID: \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"
Nov 03 21:02:43.622827 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.622696    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5qv\" (UniqueName: \"kubernetes.io/projected/95e6bf6c-083d-4286-bef8-5cb3d637cef0-kube-api-access-lh5qv\") pod \"maas-api-revoke-sub-89487c84-fsl55\" (UID: \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"
Nov 03 21:02:43.723743 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.723710    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5qv\" (UniqueName: \"kubernetes.io/projected/95e6bf6c-083d-4286-bef8-5cb3d637cef0-kube-api-access-lh5qv\") pod \"maas-api-revoke-sub-89487c84-fsl55\" (UID: \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"
Nov 03 21:02:43.723944 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.723834    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89487c84-fsl55\" (UID: \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"
Nov 03 21:02:43.723944 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.723863    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89487c84-fsl55\" (UID: \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"
Nov 03 21:02:43.724527 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.724498    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89487c84-fsl55\" (UID: \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"
Nov 03 21:02:43.726256 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.726233    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89487c84-fsl55\" (UID: \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"
Nov 03 21:02:43.736603 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.736576    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5qv\" (UniqueName: \"kubernetes.io/projected/95e6bf6c-083d-4286-bef8-5cb3d637cef0-kube-api-access-lh5qv\") pod \"maas-api-revoke-sub-89487c84-fsl55\" (UID: \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"
Nov 03 21:02:43.843926 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:43.843848    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"
Nov 03 21:02:44.119035 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:44.119008    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"]
Nov 03 21:02:44.121965 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:02:44.121937    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e6bf6c_083d_4286_bef8_5cb3d637cef0.slice/crio-a292f0a969c98b41bda8e94fec1feb0e94ec1c67d79114a3818a31c6bcb28cb0 WatchSource:0}: Error finding container a292f0a969c98b41bda8e94fec1feb0e94ec1c67d79114a3818a31c6bcb28cb0: Status 404 returned error can't find the container with id a292f0a969c98b41bda8e94fec1feb0e94ec1c67d79114a3818a31c6bcb28cb0
Nov 03 21:02:44.320530 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:44.320466    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55" event={"ID":"95e6bf6c-083d-4286-bef8-5cb3d637cef0","Type":"ContainerStarted","Data":"a292f0a969c98b41bda8e94fec1feb0e94ec1c67d79114a3818a31c6bcb28cb0"}
Nov 03 21:02:45.217054 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:02:45.217021    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:02:45.327915 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:45.327878    5659 generic.go:356] "Generic (PLEG): container finished" podID="95e6bf6c-083d-4286-bef8-5cb3d637cef0" containerID="9e6c0985b95ad469a61e4037915ce6ed076c7ef2c5971341504f4d1a8e8dd6c5" exitCode=0
Nov 03 21:02:45.328098 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:45.327964    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55" event={"ID":"95e6bf6c-083d-4286-bef8-5cb3d637cef0","Type":"ContainerDied","Data":"9e6c0985b95ad469a61e4037915ce6ed076c7ef2c5971341504f4d1a8e8dd6c5"}
Nov 03 21:02:46.474615 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:46.474592    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"
Nov 03 21:02:46.549833 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:46.549801    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-openshift-ca\") pod \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\" (UID: \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\") "
Nov 03 21:02:46.550039 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:46.549868    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e6bf6c-083d-4286-bef8-5cb3d637cef0-kube-api-access-lh5qv\" (UniqueName: \"kubernetes.io/projected/95e6bf6c-083d-4286-bef8-5cb3d637cef0-kube-api-access-lh5qv\") pod \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\" (UID: \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\") "
Nov 03 21:02:46.550039 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:46.550030    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-xks-ca\") pod \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\" (UID: \"95e6bf6c-083d-4286-bef8-5cb3d637cef0\") "
Nov 03 21:02:46.550166 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:46.550143    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-openshift-ca" pod "95e6bf6c-083d-4286-bef8-5cb3d637cef0" (UID: "95e6bf6c-083d-4286-bef8-5cb3d637cef0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:02:46.550381 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:46.550354    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:46.552061 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:46.552035    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-xks-ca" pod "95e6bf6c-083d-4286-bef8-5cb3d637cef0" (UID: "95e6bf6c-083d-4286-bef8-5cb3d637cef0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:02:46.552152 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:46.552074    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e6bf6c-083d-4286-bef8-5cb3d637cef0-kube-api-access-lh5qv" pod "95e6bf6c-083d-4286-bef8-5cb3d637cef0" (UID: "95e6bf6c-083d-4286-bef8-5cb3d637cef0"). InnerVolumeSpecName "kube-api-access-lh5qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:46.651042 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:46.650963    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e6bf6c-083d-4286-bef8-5cb3d637cef0-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:46.651042 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:46.651004    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh5qv\" (UniqueName: \"kubernetes.io/projected/95e6bf6c-083d-4286-bef8-5cb3d637cef0-kube-api-access-lh5qv\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:47.340542 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:47.340508    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"
Nov 03 21:02:47.340727 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:47.340512    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55" event={"ID":"95e6bf6c-083d-4286-bef8-5cb3d637cef0","Type":"ContainerDied","Data":"a292f0a969c98b41bda8e94fec1feb0e94ec1c67d79114a3818a31c6bcb28cb0"}
Nov 03 21:02:47.340727 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:02:47.340639    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a292f0a969c98b41bda8e94fec1feb0e94ec1c67d79114a3818a31c6bcb28cb0"
Nov 03 21:03:02.945362 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:02.945327    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:04.538909 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:04.538870    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:07.826809 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:07.826775    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:08.131822 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:08.131728    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:09.670493 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.670404    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"]
Nov 03 21:03:09.670894 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.670879    5659 cpu_manager.go:414] "containerMap: removing container" podUID="95e6bf6c-083d-4286-bef8-5cb3d637cef0" containerName="revoke-keys"
Nov 03 21:03:09.670942 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.670895    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e6bf6c-083d-4286-bef8-5cb3d637cef0" containerName="revoke-keys"
Nov 03 21:03:09.671018 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.671007    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e6bf6c-083d-4286-bef8-5cb3d637cef0" containerName="revoke-keys"
Nov 03 21:03:09.674552 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.674528    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"
Nov 03 21:03:09.680622 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.680590    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:03:09.680782 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.680712    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:03:09.681251 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.681216    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:03:09.681370 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.681250    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:03:09.685123 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.685096    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"]
Nov 03 21:03:09.820799 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.820758    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a412c5-4kf6m\" (UID: \"72553983-bbaf-4120-894a-9187ef86eed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"
Nov 03 21:03:09.820991 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.820839    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vt2c\" (UniqueName: \"kubernetes.io/projected/72553983-bbaf-4120-894a-9187ef86eed8-kube-api-access-4vt2c\") pod \"maas-api-revoke-sub-c9a412c5-4kf6m\" (UID: \"72553983-bbaf-4120-894a-9187ef86eed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"
Nov 03 21:03:09.821043 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.820977    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a412c5-4kf6m\" (UID: \"72553983-bbaf-4120-894a-9187ef86eed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"
Nov 03 21:03:09.922347 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.922255    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a412c5-4kf6m\" (UID: \"72553983-bbaf-4120-894a-9187ef86eed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"
Nov 03 21:03:09.922347 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.922320    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a412c5-4kf6m\" (UID: \"72553983-bbaf-4120-894a-9187ef86eed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"
Nov 03 21:03:09.922582 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.922383    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vt2c\" (UniqueName: \"kubernetes.io/projected/72553983-bbaf-4120-894a-9187ef86eed8-kube-api-access-4vt2c\") pod \"maas-api-revoke-sub-c9a412c5-4kf6m\" (UID: \"72553983-bbaf-4120-894a-9187ef86eed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"
Nov 03 21:03:09.923097 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.923078    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a412c5-4kf6m\" (UID: \"72553983-bbaf-4120-894a-9187ef86eed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"
Nov 03 21:03:09.924745 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.924725    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a412c5-4kf6m\" (UID: \"72553983-bbaf-4120-894a-9187ef86eed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"
Nov 03 21:03:09.935018 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.934990    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vt2c\" (UniqueName: \"kubernetes.io/projected/72553983-bbaf-4120-894a-9187ef86eed8-kube-api-access-4vt2c\") pod \"maas-api-revoke-sub-c9a412c5-4kf6m\" (UID: \"72553983-bbaf-4120-894a-9187ef86eed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"
Nov 03 21:03:09.992152 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:09.992118    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"
Nov 03 21:03:10.271840 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:10.271811    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"]
Nov 03 21:03:10.273951 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:03:10.273916    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72553983_bbaf_4120_894a_9187ef86eed8.slice/crio-fbd7b0b905e87bcb80d8308be8c531a38dc51dd04c94747893fccd1d975b59a0 WatchSource:0}: Error finding container fbd7b0b905e87bcb80d8308be8c531a38dc51dd04c94747893fccd1d975b59a0: Status 404 returned error can't find the container with id fbd7b0b905e87bcb80d8308be8c531a38dc51dd04c94747893fccd1d975b59a0
Nov 03 21:03:10.475477 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:10.475437    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m" event={"ID":"72553983-bbaf-4120-894a-9187ef86eed8","Type":"ContainerStarted","Data":"fbd7b0b905e87bcb80d8308be8c531a38dc51dd04c94747893fccd1d975b59a0"}
Nov 03 21:03:11.482712 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:11.482681    5659 generic.go:356] "Generic (PLEG): container finished" podID="72553983-bbaf-4120-894a-9187ef86eed8" containerID="3b1787def798a8e733f297a989aa19335ed13e7c3363904dc9c51ca9235c32a4" exitCode=0
Nov 03 21:03:11.483114 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:11.482770    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m" event={"ID":"72553983-bbaf-4120-894a-9187ef86eed8","Type":"ContainerDied","Data":"3b1787def798a8e733f297a989aa19335ed13e7c3363904dc9c51ca9235c32a4"}
Nov 03 21:03:12.638535 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:12.638503    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"
Nov 03 21:03:12.752225 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:12.752139    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-xks-ca\") pod \"72553983-bbaf-4120-894a-9187ef86eed8\" (UID: \"72553983-bbaf-4120-894a-9187ef86eed8\") "
Nov 03 21:03:12.752225 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:12.752192    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-openshift-ca\") pod \"72553983-bbaf-4120-894a-9187ef86eed8\" (UID: \"72553983-bbaf-4120-894a-9187ef86eed8\") "
Nov 03 21:03:12.752433 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:12.752226    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72553983-bbaf-4120-894a-9187ef86eed8-kube-api-access-4vt2c\" (UniqueName: \"kubernetes.io/projected/72553983-bbaf-4120-894a-9187ef86eed8-kube-api-access-4vt2c\") pod \"72553983-bbaf-4120-894a-9187ef86eed8\" (UID: \"72553983-bbaf-4120-894a-9187ef86eed8\") "
Nov 03 21:03:12.752559 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:12.752535    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-openshift-ca" pod "72553983-bbaf-4120-894a-9187ef86eed8" (UID: "72553983-bbaf-4120-894a-9187ef86eed8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:12.754375 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:12.754342    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-xks-ca" pod "72553983-bbaf-4120-894a-9187ef86eed8" (UID: "72553983-bbaf-4120-894a-9187ef86eed8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:03:12.754565 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:12.754470    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72553983-bbaf-4120-894a-9187ef86eed8-kube-api-access-4vt2c" pod "72553983-bbaf-4120-894a-9187ef86eed8" (UID: "72553983-bbaf-4120-894a-9187ef86eed8"). InnerVolumeSpecName "kube-api-access-4vt2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:12.853440 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:12.853403    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:12.853440 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:12.853434    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72553983-bbaf-4120-894a-9187ef86eed8-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:12.853440 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:12.853444    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vt2c\" (UniqueName: \"kubernetes.io/projected/72553983-bbaf-4120-894a-9187ef86eed8-kube-api-access-4vt2c\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:13.496528 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:13.496469    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m" event={"ID":"72553983-bbaf-4120-894a-9187ef86eed8","Type":"ContainerDied","Data":"fbd7b0b905e87bcb80d8308be8c531a38dc51dd04c94747893fccd1d975b59a0"}
Nov 03 21:03:13.496528 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:13.496516    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"
Nov 03 21:03:13.496768 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:13.496548    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbd7b0b905e87bcb80d8308be8c531a38dc51dd04c94747893fccd1d975b59a0"
Nov 03 21:03:15.216221 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:03:15.216181    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:03:19.896169 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:19.896132    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"]
Nov 03 21:03:19.898356 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:19.896600    5659 cpu_manager.go:414] "containerMap: removing container" podUID="72553983-bbaf-4120-894a-9187ef86eed8" containerName="revoke-keys"
Nov 03 21:03:19.898356 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:19.896612    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72553983-bbaf-4120-894a-9187ef86eed8" containerName="revoke-keys"
Nov 03 21:03:19.898356 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:19.896703    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="72553983-bbaf-4120-894a-9187ef86eed8" containerName="revoke-keys"
Nov 03 21:03:19.899546 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:19.899529    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"
Nov 03 21:03:19.905926 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:19.905897    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:03:19.906174 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:19.905962    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:03:19.906812 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:19.906252    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:03:19.907322 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:19.907299    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:03:19.910492 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:19.910453    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"]
Nov 03 21:03:19.919900 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:19.919859    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d9a142-5z7td\" (UID: \"bf494daf-114e-404a-8331-5820955a0c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"
Nov 03 21:03:19.920153 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:19.919960    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4zgr\" (UniqueName: \"kubernetes.io/projected/bf494daf-114e-404a-8331-5820955a0c3f-kube-api-access-b4zgr\") pod \"maas-api-revoke-sub-e3d9a142-5z7td\" (UID: \"bf494daf-114e-404a-8331-5820955a0c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"
Nov 03 21:03:19.920153 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:19.920030    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d9a142-5z7td\" (UID: \"bf494daf-114e-404a-8331-5820955a0c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"
Nov 03 21:03:20.021323 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:20.021282    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d9a142-5z7td\" (UID: \"bf494daf-114e-404a-8331-5820955a0c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"
Nov 03 21:03:20.021539 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:20.021329    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4zgr\" (UniqueName: \"kubernetes.io/projected/bf494daf-114e-404a-8331-5820955a0c3f-kube-api-access-b4zgr\") pod \"maas-api-revoke-sub-e3d9a142-5z7td\" (UID: \"bf494daf-114e-404a-8331-5820955a0c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"
Nov 03 21:03:20.021539 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:20.021367    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d9a142-5z7td\" (UID: \"bf494daf-114e-404a-8331-5820955a0c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"
Nov 03 21:03:20.021948 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:20.021913    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d9a142-5z7td\" (UID: \"bf494daf-114e-404a-8331-5820955a0c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"
Nov 03 21:03:20.023770 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:20.023749    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d9a142-5z7td\" (UID: \"bf494daf-114e-404a-8331-5820955a0c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"
Nov 03 21:03:20.034015 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:20.033986    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4zgr\" (UniqueName: \"kubernetes.io/projected/bf494daf-114e-404a-8331-5820955a0c3f-kube-api-access-b4zgr\") pod \"maas-api-revoke-sub-e3d9a142-5z7td\" (UID: \"bf494daf-114e-404a-8331-5820955a0c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"
Nov 03 21:03:20.213704 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:20.213668    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"
Nov 03 21:03:20.494812 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:20.494780    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"]
Nov 03 21:03:20.496558 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:03:20.496524    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf494daf_114e_404a_8331_5820955a0c3f.slice/crio-35ee44792a0eb006787df23833e141c6ce4fc5a31edaec11f16cc29825ae456a WatchSource:0}: Error finding container 35ee44792a0eb006787df23833e141c6ce4fc5a31edaec11f16cc29825ae456a: Status 404 returned error can't find the container with id 35ee44792a0eb006787df23833e141c6ce4fc5a31edaec11f16cc29825ae456a
Nov 03 21:03:20.535588 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:20.535555    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td" event={"ID":"bf494daf-114e-404a-8331-5820955a0c3f","Type":"ContainerStarted","Data":"35ee44792a0eb006787df23833e141c6ce4fc5a31edaec11f16cc29825ae456a"}
Nov 03 21:03:21.542835 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:21.542799    5659 generic.go:356] "Generic (PLEG): container finished" podID="bf494daf-114e-404a-8331-5820955a0c3f" containerID="6506d45ae063507f8230a25d3c04f02376b7235c31e37bff045c60aa7ebb9bdd" exitCode=0
Nov 03 21:03:21.543223 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:21.542884    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td" event={"ID":"bf494daf-114e-404a-8331-5820955a0c3f","Type":"ContainerDied","Data":"6506d45ae063507f8230a25d3c04f02376b7235c31e37bff045c60aa7ebb9bdd"}
Nov 03 21:03:22.690288 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:22.690263    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"
Nov 03 21:03:22.752509 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:22.752458    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf494daf-114e-404a-8331-5820955a0c3f-kube-api-access-b4zgr\" (UniqueName: \"kubernetes.io/projected/bf494daf-114e-404a-8331-5820955a0c3f-kube-api-access-b4zgr\") pod \"bf494daf-114e-404a-8331-5820955a0c3f\" (UID: \"bf494daf-114e-404a-8331-5820955a0c3f\") "
Nov 03 21:03:22.752694 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:22.752518    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-openshift-ca\") pod \"bf494daf-114e-404a-8331-5820955a0c3f\" (UID: \"bf494daf-114e-404a-8331-5820955a0c3f\") "
Nov 03 21:03:22.752694 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:22.752589    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-xks-ca\") pod \"bf494daf-114e-404a-8331-5820955a0c3f\" (UID: \"bf494daf-114e-404a-8331-5820955a0c3f\") "
Nov 03 21:03:22.752894 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:22.752871    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-openshift-ca" pod "bf494daf-114e-404a-8331-5820955a0c3f" (UID: "bf494daf-114e-404a-8331-5820955a0c3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:22.754615 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:22.754589    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-xks-ca" pod "bf494daf-114e-404a-8331-5820955a0c3f" (UID: "bf494daf-114e-404a-8331-5820955a0c3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:03:22.754736 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:22.754699    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf494daf-114e-404a-8331-5820955a0c3f-kube-api-access-b4zgr" pod "bf494daf-114e-404a-8331-5820955a0c3f" (UID: "bf494daf-114e-404a-8331-5820955a0c3f"). InnerVolumeSpecName "kube-api-access-b4zgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:22.853455 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:22.853366    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4zgr\" (UniqueName: \"kubernetes.io/projected/bf494daf-114e-404a-8331-5820955a0c3f-kube-api-access-b4zgr\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:22.853455 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:22.853403    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:22.853455 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:22.853414    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf494daf-114e-404a-8331-5820955a0c3f-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:23.555260 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:23.555228    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"
Nov 03 21:03:23.555449 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:23.555227    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td" event={"ID":"bf494daf-114e-404a-8331-5820955a0c3f","Type":"ContainerDied","Data":"35ee44792a0eb006787df23833e141c6ce4fc5a31edaec11f16cc29825ae456a"}
Nov 03 21:03:23.555449 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:23.555353    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35ee44792a0eb006787df23833e141c6ce4fc5a31edaec11f16cc29825ae456a"
Nov 03 21:03:28.584945 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:28.584910    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:03:45.216886 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:03:45.216847    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:03:47.033499 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.033455    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"]
Nov 03 21:03:47.033909 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.033896    5659 cpu_manager.go:414] "containerMap: removing container" podUID="bf494daf-114e-404a-8331-5820955a0c3f" containerName="revoke-keys"
Nov 03 21:03:47.033960 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.033910    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf494daf-114e-404a-8331-5820955a0c3f" containerName="revoke-keys"
Nov 03 21:03:47.034006 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.033998    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf494daf-114e-404a-8331-5820955a0c3f" containerName="revoke-keys"
Nov 03 21:03:47.037133 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.037116    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"
Nov 03 21:03:47.043375 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.043349    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:03:47.043588 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.043570    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:03:47.043663 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.043637    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:03:47.043992 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.043963    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:03:47.047328 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.047306    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"]
Nov 03 21:03:47.126966 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.126924    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bce983d-sbgcs\" (UID: \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"
Nov 03 21:03:47.127158 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.126974    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bce983d-sbgcs\" (UID: \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"
Nov 03 21:03:47.127158 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.127096    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cjrs\" (UniqueName: \"kubernetes.io/projected/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-kube-api-access-2cjrs\") pod \"maas-api-revoke-sub-2bce983d-sbgcs\" (UID: \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"
Nov 03 21:03:47.228865 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.228824    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bce983d-sbgcs\" (UID: \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"
Nov 03 21:03:47.229154 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.229131    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bce983d-sbgcs\" (UID: \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"
Nov 03 21:03:47.229334 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.229307    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cjrs\" (UniqueName: \"kubernetes.io/projected/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-kube-api-access-2cjrs\") pod \"maas-api-revoke-sub-2bce983d-sbgcs\" (UID: \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"
Nov 03 21:03:47.230338 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.230310    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bce983d-sbgcs\" (UID: \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"
Nov 03 21:03:47.232116 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.232091    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bce983d-sbgcs\" (UID: \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"
Nov 03 21:03:47.241542 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.241513    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cjrs\" (UniqueName: \"kubernetes.io/projected/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-kube-api-access-2cjrs\") pod \"maas-api-revoke-sub-2bce983d-sbgcs\" (UID: \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"
Nov 03 21:03:47.352764 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.352676    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"
Nov 03 21:03:47.634513 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.634468    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"]
Nov 03 21:03:47.636599 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:03:47.636572    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb5dbb1_5893_4c22_a76e_ea24fda0b78b.slice/crio-d5ba4540f3f46e0f315917e8421aa0cb632705f41fc3efee8cac44dfc209e199 WatchSource:0}: Error finding container d5ba4540f3f46e0f315917e8421aa0cb632705f41fc3efee8cac44dfc209e199: Status 404 returned error can't find the container with id d5ba4540f3f46e0f315917e8421aa0cb632705f41fc3efee8cac44dfc209e199
Nov 03 21:03:47.716577 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:47.716540    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs" event={"ID":"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b","Type":"ContainerStarted","Data":"d5ba4540f3f46e0f315917e8421aa0cb632705f41fc3efee8cac44dfc209e199"}
Nov 03 21:03:48.723735 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:48.723697    5659 generic.go:356] "Generic (PLEG): container finished" podID="9eb5dbb1-5893-4c22-a76e-ea24fda0b78b" containerID="e8ff34d1c0a4a3fb031f7a3d8d96396d592d01adee64d6d18644eba3c0c659b5" exitCode=0
Nov 03 21:03:48.724149 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:48.723751    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs" event={"ID":"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b","Type":"ContainerDied","Data":"e8ff34d1c0a4a3fb031f7a3d8d96396d592d01adee64d6d18644eba3c0c659b5"}
Nov 03 21:03:49.873221 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:49.873193    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"
Nov 03 21:03:49.956453 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:49.956417    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-kube-api-access-2cjrs\" (UniqueName: \"kubernetes.io/projected/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-kube-api-access-2cjrs\") pod \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\" (UID: \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\") "
Nov 03 21:03:49.956667 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:49.956515    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-openshift-ca\") pod \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\" (UID: \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\") "
Nov 03 21:03:49.956667 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:49.956636    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-xks-ca\") pod \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\" (UID: \"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b\") "
Nov 03 21:03:49.956939 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:49.956910    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-openshift-ca" pod "9eb5dbb1-5893-4c22-a76e-ea24fda0b78b" (UID: "9eb5dbb1-5893-4c22-a76e-ea24fda0b78b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:49.958664 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:49.958634    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-xks-ca" pod "9eb5dbb1-5893-4c22-a76e-ea24fda0b78b" (UID: "9eb5dbb1-5893-4c22-a76e-ea24fda0b78b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:03:49.958775 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:49.958678    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-kube-api-access-2cjrs" pod "9eb5dbb1-5893-4c22-a76e-ea24fda0b78b" (UID: "9eb5dbb1-5893-4c22-a76e-ea24fda0b78b"). InnerVolumeSpecName "kube-api-access-2cjrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:50.057709 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:50.057615    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.057709 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:50.057652    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cjrs\" (UniqueName: \"kubernetes.io/projected/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-kube-api-access-2cjrs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.057709 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:50.057663    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eb5dbb1-5893-4c22-a76e-ea24fda0b78b-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:50.736394 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:50.736356    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs" event={"ID":"9eb5dbb1-5893-4c22-a76e-ea24fda0b78b","Type":"ContainerDied","Data":"d5ba4540f3f46e0f315917e8421aa0cb632705f41fc3efee8cac44dfc209e199"}
Nov 03 21:03:50.736602 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:50.736397    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"
Nov 03 21:03:50.736602 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:50.736438    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5ba4540f3f46e0f315917e8421aa0cb632705f41fc3efee8cac44dfc209e199"
Nov 03 21:03:57.253651 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.253612    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"]
Nov 03 21:03:57.254174 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.254148    5659 cpu_manager.go:414] "containerMap: removing container" podUID="9eb5dbb1-5893-4c22-a76e-ea24fda0b78b" containerName="revoke-keys"
Nov 03 21:03:57.254174 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.254163    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb5dbb1-5893-4c22-a76e-ea24fda0b78b" containerName="revoke-keys"
Nov 03 21:03:57.254305 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.254249    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb5dbb1-5893-4c22-a76e-ea24fda0b78b" containerName="revoke-keys"
Nov 03 21:03:57.258158 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.258132    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"
Nov 03 21:03:57.263559 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.263534    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:03:57.263747 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.263534    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:03:57.263747 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.263557    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:03:57.263915 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.263658    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:03:57.267318 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.267291    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"]
Nov 03 21:03:57.331701 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.331657    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73d6ee39-hc2dv\" (UID: \"72714747-8a61-43b2-b2d3-01e5b7157150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"
Nov 03 21:03:57.331898 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.331780    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmq2\" (UniqueName: \"kubernetes.io/projected/72714747-8a61-43b2-b2d3-01e5b7157150-kube-api-access-hhmq2\") pod \"maas-api-revoke-sub-73d6ee39-hc2dv\" (UID: \"72714747-8a61-43b2-b2d3-01e5b7157150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"
Nov 03 21:03:57.331898 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.331822    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73d6ee39-hc2dv\" (UID: \"72714747-8a61-43b2-b2d3-01e5b7157150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"
Nov 03 21:03:57.432765 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.432722    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73d6ee39-hc2dv\" (UID: \"72714747-8a61-43b2-b2d3-01e5b7157150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"
Nov 03 21:03:57.432982 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.432783    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmq2\" (UniqueName: \"kubernetes.io/projected/72714747-8a61-43b2-b2d3-01e5b7157150-kube-api-access-hhmq2\") pod \"maas-api-revoke-sub-73d6ee39-hc2dv\" (UID: \"72714747-8a61-43b2-b2d3-01e5b7157150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"
Nov 03 21:03:57.432982 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.432823    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73d6ee39-hc2dv\" (UID: \"72714747-8a61-43b2-b2d3-01e5b7157150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"
Nov 03 21:03:57.433592 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.433570    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73d6ee39-hc2dv\" (UID: \"72714747-8a61-43b2-b2d3-01e5b7157150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"
Nov 03 21:03:57.435241 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.435220    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73d6ee39-hc2dv\" (UID: \"72714747-8a61-43b2-b2d3-01e5b7157150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"
Nov 03 21:03:57.445225 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.445200    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmq2\" (UniqueName: \"kubernetes.io/projected/72714747-8a61-43b2-b2d3-01e5b7157150-kube-api-access-hhmq2\") pod \"maas-api-revoke-sub-73d6ee39-hc2dv\" (UID: \"72714747-8a61-43b2-b2d3-01e5b7157150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"
Nov 03 21:03:57.575237 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.575149    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"
Nov 03 21:03:57.852998 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:57.852972    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"]
Nov 03 21:03:57.854671 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:03:57.854644    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72714747_8a61_43b2_b2d3_01e5b7157150.slice/crio-29730f23636f53372154cbc6cf0c24974d9f5e73a65f620b4854802e62dc8e37 WatchSource:0}: Error finding container 29730f23636f53372154cbc6cf0c24974d9f5e73a65f620b4854802e62dc8e37: Status 404 returned error can't find the container with id 29730f23636f53372154cbc6cf0c24974d9f5e73a65f620b4854802e62dc8e37
Nov 03 21:03:58.783960 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:58.783921    5659 generic.go:356] "Generic (PLEG): container finished" podID="72714747-8a61-43b2-b2d3-01e5b7157150" containerID="a9aebcc379a078bf125dbc6d75149b31564408e1866934a9f3e0607d4cddcc39" exitCode=0
Nov 03 21:03:58.784429 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:58.784004    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv" event={"ID":"72714747-8a61-43b2-b2d3-01e5b7157150","Type":"ContainerDied","Data":"a9aebcc379a078bf125dbc6d75149b31564408e1866934a9f3e0607d4cddcc39"}
Nov 03 21:03:58.784429 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:03:58.784068    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv" event={"ID":"72714747-8a61-43b2-b2d3-01e5b7157150","Type":"ContainerStarted","Data":"29730f23636f53372154cbc6cf0c24974d9f5e73a65f620b4854802e62dc8e37"}
Nov 03 21:04:00.005131 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:00.004875    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"
Nov 03 21:04:00.088363 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:00.088248    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-xks-ca\") pod \"72714747-8a61-43b2-b2d3-01e5b7157150\" (UID: \"72714747-8a61-43b2-b2d3-01e5b7157150\") "
Nov 03 21:04:00.088363 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:00.088346    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72714747-8a61-43b2-b2d3-01e5b7157150-kube-api-access-hhmq2\" (UniqueName: \"kubernetes.io/projected/72714747-8a61-43b2-b2d3-01e5b7157150-kube-api-access-hhmq2\") pod \"72714747-8a61-43b2-b2d3-01e5b7157150\" (UID: \"72714747-8a61-43b2-b2d3-01e5b7157150\") "
Nov 03 21:04:00.088656 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:00.088436    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-openshift-ca\") pod \"72714747-8a61-43b2-b2d3-01e5b7157150\" (UID: \"72714747-8a61-43b2-b2d3-01e5b7157150\") "
Nov 03 21:04:00.090260 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:00.090188    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-openshift-ca" pod "72714747-8a61-43b2-b2d3-01e5b7157150" (UID: "72714747-8a61-43b2-b2d3-01e5b7157150"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:00.094928 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:00.094889    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-xks-ca" pod "72714747-8a61-43b2-b2d3-01e5b7157150" (UID: "72714747-8a61-43b2-b2d3-01e5b7157150"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:00.096091 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:00.096067    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72714747-8a61-43b2-b2d3-01e5b7157150-kube-api-access-hhmq2" pod "72714747-8a61-43b2-b2d3-01e5b7157150" (UID: "72714747-8a61-43b2-b2d3-01e5b7157150"). InnerVolumeSpecName "kube-api-access-hhmq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:00.190546 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:00.190476    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:00.190546 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:00.190548    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72714747-8a61-43b2-b2d3-01e5b7157150-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:00.190845 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:00.190567    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhmq2\" (UniqueName: \"kubernetes.io/projected/72714747-8a61-43b2-b2d3-01e5b7157150-kube-api-access-hhmq2\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:00.801755 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:00.801718    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv"
Nov 03 21:04:00.801755 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:00.801739    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d6ee39-hc2dv" event={"ID":"72714747-8a61-43b2-b2d3-01e5b7157150","Type":"ContainerDied","Data":"29730f23636f53372154cbc6cf0c24974d9f5e73a65f620b4854802e62dc8e37"}
Nov 03 21:04:00.801999 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:00.801786    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29730f23636f53372154cbc6cf0c24974d9f5e73a65f620b4854802e62dc8e37"
Nov 03 21:04:07.880370 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:07.880335    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:08.188160 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:08.188074    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:15.216906 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:04:15.216867    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:04:27.433852 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.433816    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"]
Nov 03 21:04:27.434347 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.434295    5659 cpu_manager.go:414] "containerMap: removing container" podUID="72714747-8a61-43b2-b2d3-01e5b7157150" containerName="revoke-keys"
Nov 03 21:04:27.434347 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.434312    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72714747-8a61-43b2-b2d3-01e5b7157150" containerName="revoke-keys"
Nov 03 21:04:27.434471 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.434438    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="72714747-8a61-43b2-b2d3-01e5b7157150" containerName="revoke-keys"
Nov 03 21:04:27.442801 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.442771    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"
Nov 03 21:04:27.447068 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.447033    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"]
Nov 03 21:04:27.447219 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.447150    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:04:27.448042 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.448017    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:04:27.448162 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.448071    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:04:27.448162 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.448128    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:04:27.586606 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.586565    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a27a9d3-fjltl\" (UID: \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"
Nov 03 21:04:27.586802 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.586638    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tr9f\" (UniqueName: \"kubernetes.io/projected/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-kube-api-access-4tr9f\") pod \"maas-api-revoke-sub-7a27a9d3-fjltl\" (UID: \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"
Nov 03 21:04:27.586802 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.586730    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a27a9d3-fjltl\" (UID: \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"
Nov 03 21:04:27.688353 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.688263    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tr9f\" (UniqueName: \"kubernetes.io/projected/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-kube-api-access-4tr9f\") pod \"maas-api-revoke-sub-7a27a9d3-fjltl\" (UID: \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"
Nov 03 21:04:27.688353 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.688323    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a27a9d3-fjltl\" (UID: \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"
Nov 03 21:04:27.688602 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.688399    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a27a9d3-fjltl\" (UID: \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"
Nov 03 21:04:27.688984 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.688951    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a27a9d3-fjltl\" (UID: \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"
Nov 03 21:04:27.690718 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.690700    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a27a9d3-fjltl\" (UID: \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"
Nov 03 21:04:27.701179 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.701155    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tr9f\" (UniqueName: \"kubernetes.io/projected/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-kube-api-access-4tr9f\") pod \"maas-api-revoke-sub-7a27a9d3-fjltl\" (UID: \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"
Nov 03 21:04:27.761249 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:27.761218    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"
Nov 03 21:04:28.040098 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:28.040066    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"]
Nov 03 21:04:28.041700 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:04:28.041673    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8024e4b_e3af_4bc5_935f_dddc089e0ed1.slice/crio-ca55edbd4ea5a4e5306d46d9087137fc674e9967c850ed38c2cc52b5ccc37d5d WatchSource:0}: Error finding container ca55edbd4ea5a4e5306d46d9087137fc674e9967c850ed38c2cc52b5ccc37d5d: Status 404 returned error can't find the container with id ca55edbd4ea5a4e5306d46d9087137fc674e9967c850ed38c2cc52b5ccc37d5d
Nov 03 21:04:28.978829 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:28.978783    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl" event={"ID":"e8024e4b-e3af-4bc5-935f-dddc089e0ed1","Type":"ContainerStarted","Data":"0c79fbe676f4110342b9b2c612f2625d89317483d145c09e24e287d13f103b22"}
Nov 03 21:04:28.979216 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:28.978878    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl" event={"ID":"e8024e4b-e3af-4bc5-935f-dddc089e0ed1","Type":"ContainerStarted","Data":"ca55edbd4ea5a4e5306d46d9087137fc674e9967c850ed38c2cc52b5ccc37d5d"}
Nov 03 21:04:28.999691 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:28.999643    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl" podStartSLOduration=1.9996287700000002 podStartE2EDuration="1.99962877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:28.997186731 +0000 UTC m=+5221.152089153" watchObservedRunningTime="2026-11-03 21:04:28.99962877 +0000 UTC m=+5221.154531191"
Nov 03 21:04:29.985500 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:29.985440    5659 generic.go:356] "Generic (PLEG): container finished" podID="e8024e4b-e3af-4bc5-935f-dddc089e0ed1" containerID="0c79fbe676f4110342b9b2c612f2625d89317483d145c09e24e287d13f103b22" exitCode=0
Nov 03 21:04:29.985932 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:29.985519    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl" event={"ID":"e8024e4b-e3af-4bc5-935f-dddc089e0ed1","Type":"ContainerDied","Data":"0c79fbe676f4110342b9b2c612f2625d89317483d145c09e24e287d13f103b22"}
Nov 03 21:04:31.128510 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.128468    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"
Nov 03 21:04:31.226045 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.226011    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-kube-api-access-4tr9f\" (UniqueName: \"kubernetes.io/projected/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-kube-api-access-4tr9f\") pod \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\" (UID: \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\") "
Nov 03 21:04:31.226248 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.226067    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-openshift-ca\") pod \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\" (UID: \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\") "
Nov 03 21:04:31.226248 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.226120    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-xks-ca\") pod \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\" (UID: \"e8024e4b-e3af-4bc5-935f-dddc089e0ed1\") "
Nov 03 21:04:31.226446 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.226423    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-openshift-ca" pod "e8024e4b-e3af-4bc5-935f-dddc089e0ed1" (UID: "e8024e4b-e3af-4bc5-935f-dddc089e0ed1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:31.228184 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.228159    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-xks-ca" pod "e8024e4b-e3af-4bc5-935f-dddc089e0ed1" (UID: "e8024e4b-e3af-4bc5-935f-dddc089e0ed1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:31.228297 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.228273    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-kube-api-access-4tr9f" pod "e8024e4b-e3af-4bc5-935f-dddc089e0ed1" (UID: "e8024e4b-e3af-4bc5-935f-dddc089e0ed1"). InnerVolumeSpecName "kube-api-access-4tr9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:31.327788 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.327712    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:31.327788 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.327741    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tr9f\" (UniqueName: \"kubernetes.io/projected/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-kube-api-access-4tr9f\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:31.327788 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.327753    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8024e4b-e3af-4bc5-935f-dddc089e0ed1-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:31.400993 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.400961    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:31.533734 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.533693    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:31.999156 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.999127    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl"
Nov 03 21:04:31.999333 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.999132    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a27a9d3-fjltl" event={"ID":"e8024e4b-e3af-4bc5-935f-dddc089e0ed1","Type":"ContainerDied","Data":"ca55edbd4ea5a4e5306d46d9087137fc674e9967c850ed38c2cc52b5ccc37d5d"}
Nov 03 21:04:31.999333 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:31.999263    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca55edbd4ea5a4e5306d46d9087137fc674e9967c850ed38c2cc52b5ccc37d5d"
Nov 03 21:04:37.650469 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.650418    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"]
Nov 03 21:04:37.651012 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.650902    5659 cpu_manager.go:414] "containerMap: removing container" podUID="e8024e4b-e3af-4bc5-935f-dddc089e0ed1" containerName="revoke-keys"
Nov 03 21:04:37.651012 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.650915    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8024e4b-e3af-4bc5-935f-dddc089e0ed1" containerName="revoke-keys"
Nov 03 21:04:37.651150 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.651015    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8024e4b-e3af-4bc5-935f-dddc089e0ed1" containerName="revoke-keys"
Nov 03 21:04:37.654550 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.654524    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"
Nov 03 21:04:37.658934 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.658906    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:04:37.660074 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.660050    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:04:37.664090 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.664063    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"]
Nov 03 21:04:37.671454 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.671428    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:04:37.671454 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.671427    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:04:37.795393 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.795354    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpzj\" (UniqueName: \"kubernetes.io/projected/508f00aa-c3ac-436a-a618-8ec2ec8362ed-kube-api-access-8dpzj\") pod \"maas-api-revoke-sub-1087b164-jchw2\" (UID: \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"
Nov 03 21:04:37.795625 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.795415    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1087b164-jchw2\" (UID: \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"
Nov 03 21:04:37.795625 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.795506    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1087b164-jchw2\" (UID: \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"
Nov 03 21:04:37.897109 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.897065    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpzj\" (UniqueName: \"kubernetes.io/projected/508f00aa-c3ac-436a-a618-8ec2ec8362ed-kube-api-access-8dpzj\") pod \"maas-api-revoke-sub-1087b164-jchw2\" (UID: \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"
Nov 03 21:04:37.897287 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.897130    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1087b164-jchw2\" (UID: \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"
Nov 03 21:04:37.897287 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.897150    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1087b164-jchw2\" (UID: \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"
Nov 03 21:04:37.897854 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.897826    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1087b164-jchw2\" (UID: \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"
Nov 03 21:04:37.899550 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.899524    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1087b164-jchw2\" (UID: \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"
Nov 03 21:04:37.909326 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.909274    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpzj\" (UniqueName: \"kubernetes.io/projected/508f00aa-c3ac-436a-a618-8ec2ec8362ed-kube-api-access-8dpzj\") pod \"maas-api-revoke-sub-1087b164-jchw2\" (UID: \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"
Nov 03 21:04:37.969675 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:37.969641    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"
Nov 03 21:04:38.247708 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:38.247683    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"]
Nov 03 21:04:38.249525 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:04:38.249494    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508f00aa_c3ac_436a_a618_8ec2ec8362ed.slice/crio-0432eaf869a4466035ff13109d3fa15e3f77c900abb9a9d1927fd044ed43efea WatchSource:0}: Error finding container 0432eaf869a4466035ff13109d3fa15e3f77c900abb9a9d1927fd044ed43efea: Status 404 returned error can't find the container with id 0432eaf869a4466035ff13109d3fa15e3f77c900abb9a9d1927fd044ed43efea
Nov 03 21:04:39.044707 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:39.044671    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2" event={"ID":"508f00aa-c3ac-436a-a618-8ec2ec8362ed","Type":"ContainerStarted","Data":"30ee1035c85ca263d5312e56ea7f361a52978dc81036c4474450a244d98f8145"}
Nov 03 21:04:39.045145 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:39.044733    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2" event={"ID":"508f00aa-c3ac-436a-a618-8ec2ec8362ed","Type":"ContainerStarted","Data":"0432eaf869a4466035ff13109d3fa15e3f77c900abb9a9d1927fd044ed43efea"}
Nov 03 21:04:39.069313 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:39.069264    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2" podStartSLOduration=2.069250767 podStartE2EDuration="2.069250767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:39.067106587 +0000 UTC m=+5231.222009007" watchObservedRunningTime="2026-11-03 21:04:39.069250767 +0000 UTC m=+5231.224153186"
Nov 03 21:04:40.054241 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:40.054206    5659 generic.go:356] "Generic (PLEG): container finished" podID="508f00aa-c3ac-436a-a618-8ec2ec8362ed" containerID="30ee1035c85ca263d5312e56ea7f361a52978dc81036c4474450a244d98f8145" exitCode=0
Nov 03 21:04:40.054695 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:40.054291    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2" event={"ID":"508f00aa-c3ac-436a-a618-8ec2ec8362ed","Type":"ContainerDied","Data":"30ee1035c85ca263d5312e56ea7f361a52978dc81036c4474450a244d98f8145"}
Nov 03 21:04:41.270753 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:41.270715    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"
Nov 03 21:04:41.330802 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:41.330704    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508f00aa-c3ac-436a-a618-8ec2ec8362ed-kube-api-access-8dpzj\" (UniqueName: \"kubernetes.io/projected/508f00aa-c3ac-436a-a618-8ec2ec8362ed-kube-api-access-8dpzj\") pod \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\" (UID: \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\") "
Nov 03 21:04:41.331005 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:41.330819    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-xks-ca\") pod \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\" (UID: \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\") "
Nov 03 21:04:41.331005 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:41.330879    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-openshift-ca\") pod \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\" (UID: \"508f00aa-c3ac-436a-a618-8ec2ec8362ed\") "
Nov 03 21:04:41.331287 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:41.331259    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-openshift-ca" pod "508f00aa-c3ac-436a-a618-8ec2ec8362ed" (UID: "508f00aa-c3ac-436a-a618-8ec2ec8362ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:41.340615 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:41.340581    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-xks-ca" pod "508f00aa-c3ac-436a-a618-8ec2ec8362ed" (UID: "508f00aa-c3ac-436a-a618-8ec2ec8362ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:41.359271 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:41.359204    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508f00aa-c3ac-436a-a618-8ec2ec8362ed-kube-api-access-8dpzj" pod "508f00aa-c3ac-436a-a618-8ec2ec8362ed" (UID: "508f00aa-c3ac-436a-a618-8ec2ec8362ed"). InnerVolumeSpecName "kube-api-access-8dpzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:41.431828 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:41.431786    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:41.431828 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:41.431821    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/508f00aa-c3ac-436a-a618-8ec2ec8362ed-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:41.431828 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:41.431833    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dpzj\" (UniqueName: \"kubernetes.io/projected/508f00aa-c3ac-436a-a618-8ec2ec8362ed-kube-api-access-8dpzj\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:42.070969 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:42.070939    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2"
Nov 03 21:04:42.071141 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:42.070970    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1087b164-jchw2" event={"ID":"508f00aa-c3ac-436a-a618-8ec2ec8362ed","Type":"ContainerDied","Data":"0432eaf869a4466035ff13109d3fa15e3f77c900abb9a9d1927fd044ed43efea"}
Nov 03 21:04:42.071141 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:42.071020    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0432eaf869a4466035ff13109d3fa15e3f77c900abb9a9d1927fd044ed43efea"
Nov 03 21:04:45.216981 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:04:45.216942    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:04:45.283544 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:45.283515    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:45.437052 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:45.437016    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:04:47.854466 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.854433    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"]
Nov 03 21:04:47.854918 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.854903    5659 cpu_manager.go:414] "containerMap: removing container" podUID="508f00aa-c3ac-436a-a618-8ec2ec8362ed" containerName="revoke-keys"
Nov 03 21:04:47.854965 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.854919    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508f00aa-c3ac-436a-a618-8ec2ec8362ed" containerName="revoke-keys"
Nov 03 21:04:47.855018 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.855008    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="508f00aa-c3ac-436a-a618-8ec2ec8362ed" containerName="revoke-keys"
Nov 03 21:04:47.858269 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.858248    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"
Nov 03 21:04:47.864842 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.864811    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:04:47.865061 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.864821    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:04:47.865061 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.864824    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:04:47.865263 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.864879    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:04:47.868904 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.868880    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"]
Nov 03 21:04:47.890254 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.890223    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a82f982-m2w9g\" (UID: \"74fa04c3-5af3-4f34-bb22-2832cee2accc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"
Nov 03 21:04:47.890436 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.890282    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a82f982-m2w9g\" (UID: \"74fa04c3-5af3-4f34-bb22-2832cee2accc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"
Nov 03 21:04:47.890436 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.890409    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwnjj\" (UniqueName: \"kubernetes.io/projected/74fa04c3-5af3-4f34-bb22-2832cee2accc-kube-api-access-gwnjj\") pod \"maas-api-revoke-sub-7a82f982-m2w9g\" (UID: \"74fa04c3-5af3-4f34-bb22-2832cee2accc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"
Nov 03 21:04:47.991564 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.991531    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a82f982-m2w9g\" (UID: \"74fa04c3-5af3-4f34-bb22-2832cee2accc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"
Nov 03 21:04:47.991758 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.991593    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a82f982-m2w9g\" (UID: \"74fa04c3-5af3-4f34-bb22-2832cee2accc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"
Nov 03 21:04:47.991758 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.991685    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwnjj\" (UniqueName: \"kubernetes.io/projected/74fa04c3-5af3-4f34-bb22-2832cee2accc-kube-api-access-gwnjj\") pod \"maas-api-revoke-sub-7a82f982-m2w9g\" (UID: \"74fa04c3-5af3-4f34-bb22-2832cee2accc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"
Nov 03 21:04:47.992252 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.992225    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a82f982-m2w9g\" (UID: \"74fa04c3-5af3-4f34-bb22-2832cee2accc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"
Nov 03 21:04:47.993909 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:47.993892    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a82f982-m2w9g\" (UID: \"74fa04c3-5af3-4f34-bb22-2832cee2accc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"
Nov 03 21:04:48.004285 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:48.004259    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwnjj\" (UniqueName: \"kubernetes.io/projected/74fa04c3-5af3-4f34-bb22-2832cee2accc-kube-api-access-gwnjj\") pod \"maas-api-revoke-sub-7a82f982-m2w9g\" (UID: \"74fa04c3-5af3-4f34-bb22-2832cee2accc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"
Nov 03 21:04:48.172850 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:48.172775    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"
Nov 03 21:04:48.447518 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:48.447463    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"]
Nov 03 21:04:48.448928 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:04:48.448894    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fa04c3_5af3_4f34_bb22_2832cee2accc.slice/crio-ae0177a7857f9d861d43cc301ef6ed145b38d34b4b5eb6066257139e3ded2663 WatchSource:0}: Error finding container ae0177a7857f9d861d43cc301ef6ed145b38d34b4b5eb6066257139e3ded2663: Status 404 returned error can't find the container with id ae0177a7857f9d861d43cc301ef6ed145b38d34b4b5eb6066257139e3ded2663
Nov 03 21:04:49.111156 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:49.111062    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g" event={"ID":"74fa04c3-5af3-4f34-bb22-2832cee2accc","Type":"ContainerStarted","Data":"ac840fa3676b7ca36141e3255a5a417554d65acd0e1017343154db262243f5a8"}
Nov 03 21:04:49.111156 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:49.111127    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g" event={"ID":"74fa04c3-5af3-4f34-bb22-2832cee2accc","Type":"ContainerStarted","Data":"ae0177a7857f9d861d43cc301ef6ed145b38d34b4b5eb6066257139e3ded2663"}
Nov 03 21:04:49.130317 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:49.130268    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g" podStartSLOduration=2.130256209 podStartE2EDuration="2.130256209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:49.12837768 +0000 UTC m=+5241.283280116" watchObservedRunningTime="2026-11-03 21:04:49.130256209 +0000 UTC m=+5241.285158628"
Nov 03 21:04:50.118333 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:50.118298    5659 generic.go:356] "Generic (PLEG): container finished" podID="74fa04c3-5af3-4f34-bb22-2832cee2accc" containerID="ac840fa3676b7ca36141e3255a5a417554d65acd0e1017343154db262243f5a8" exitCode=0
Nov 03 21:04:50.118770 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:50.118370    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g" event={"ID":"74fa04c3-5af3-4f34-bb22-2832cee2accc","Type":"ContainerDied","Data":"ac840fa3676b7ca36141e3255a5a417554d65acd0e1017343154db262243f5a8"}
Nov 03 21:04:51.263143 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:51.263119    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"
Nov 03 21:04:51.326947 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:51.326908    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-openshift-ca\") pod \"74fa04c3-5af3-4f34-bb22-2832cee2accc\" (UID: \"74fa04c3-5af3-4f34-bb22-2832cee2accc\") "
Nov 03 21:04:51.327134 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:51.326997    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-xks-ca\") pod \"74fa04c3-5af3-4f34-bb22-2832cee2accc\" (UID: \"74fa04c3-5af3-4f34-bb22-2832cee2accc\") "
Nov 03 21:04:51.327134 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:51.327041    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74fa04c3-5af3-4f34-bb22-2832cee2accc-kube-api-access-gwnjj\" (UniqueName: \"kubernetes.io/projected/74fa04c3-5af3-4f34-bb22-2832cee2accc-kube-api-access-gwnjj\") pod \"74fa04c3-5af3-4f34-bb22-2832cee2accc\" (UID: \"74fa04c3-5af3-4f34-bb22-2832cee2accc\") "
Nov 03 21:04:51.327359 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:51.327337    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-openshift-ca" pod "74fa04c3-5af3-4f34-bb22-2832cee2accc" (UID: "74fa04c3-5af3-4f34-bb22-2832cee2accc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:51.329179 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:51.329154    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-xks-ca" pod "74fa04c3-5af3-4f34-bb22-2832cee2accc" (UID: "74fa04c3-5af3-4f34-bb22-2832cee2accc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:51.329290 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:51.329156    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fa04c3-5af3-4f34-bb22-2832cee2accc-kube-api-access-gwnjj" pod "74fa04c3-5af3-4f34-bb22-2832cee2accc" (UID: "74fa04c3-5af3-4f34-bb22-2832cee2accc"). InnerVolumeSpecName "kube-api-access-gwnjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:51.427893 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:51.427809    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:51.427893 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:51.427841    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74fa04c3-5af3-4f34-bb22-2832cee2accc-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:51.427893 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:51.427851    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwnjj\" (UniqueName: \"kubernetes.io/projected/74fa04c3-5af3-4f34-bb22-2832cee2accc-kube-api-access-gwnjj\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:52.131075 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:52.131043    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g"
Nov 03 21:04:52.131229 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:52.131038    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a82f982-m2w9g" event={"ID":"74fa04c3-5af3-4f34-bb22-2832cee2accc","Type":"ContainerDied","Data":"ae0177a7857f9d861d43cc301ef6ed145b38d34b4b5eb6066257139e3ded2663"}
Nov 03 21:04:52.131229 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:52.131177    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae0177a7857f9d861d43cc301ef6ed145b38d34b4b5eb6066257139e3ded2663"
Nov 03 21:04:58.073745 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.073705    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"]
Nov 03 21:04:58.074215 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.074199    5659 cpu_manager.go:414] "containerMap: removing container" podUID="74fa04c3-5af3-4f34-bb22-2832cee2accc" containerName="revoke-keys"
Nov 03 21:04:58.074270 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.074218    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74fa04c3-5af3-4f34-bb22-2832cee2accc" containerName="revoke-keys"
Nov 03 21:04:58.074328 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.074316    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="74fa04c3-5af3-4f34-bb22-2832cee2accc" containerName="revoke-keys"
Nov 03 21:04:58.077810 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.077788    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"
Nov 03 21:04:58.084302 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.084272    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:04:58.084456 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.084338    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:04:58.084456 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.084357    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:04:58.084456 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.084275    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:04:58.086526 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.086503    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"]
Nov 03 21:04:58.200107 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.200072    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15c6a7c6-8b65k\" (UID: \"3c0c479d-7754-4d33-bdac-72ff5f82900e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"
Nov 03 21:04:58.200107 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.200110    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15c6a7c6-8b65k\" (UID: \"3c0c479d-7754-4d33-bdac-72ff5f82900e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"
Nov 03 21:04:58.200338 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.200147    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqcs\" (UniqueName: \"kubernetes.io/projected/3c0c479d-7754-4d33-bdac-72ff5f82900e-kube-api-access-jhqcs\") pod \"maas-api-revoke-sub-15c6a7c6-8b65k\" (UID: \"3c0c479d-7754-4d33-bdac-72ff5f82900e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"
Nov 03 21:04:58.301161 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.301122    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15c6a7c6-8b65k\" (UID: \"3c0c479d-7754-4d33-bdac-72ff5f82900e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"
Nov 03 21:04:58.301161 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.301165    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15c6a7c6-8b65k\" (UID: \"3c0c479d-7754-4d33-bdac-72ff5f82900e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"
Nov 03 21:04:58.301403 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.301193    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqcs\" (UniqueName: \"kubernetes.io/projected/3c0c479d-7754-4d33-bdac-72ff5f82900e-kube-api-access-jhqcs\") pod \"maas-api-revoke-sub-15c6a7c6-8b65k\" (UID: \"3c0c479d-7754-4d33-bdac-72ff5f82900e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"
Nov 03 21:04:58.301952 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.301930    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15c6a7c6-8b65k\" (UID: \"3c0c479d-7754-4d33-bdac-72ff5f82900e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"
Nov 03 21:04:58.303575 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.303554    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15c6a7c6-8b65k\" (UID: \"3c0c479d-7754-4d33-bdac-72ff5f82900e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"
Nov 03 21:04:58.313856 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.313833    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqcs\" (UniqueName: \"kubernetes.io/projected/3c0c479d-7754-4d33-bdac-72ff5f82900e-kube-api-access-jhqcs\") pod \"maas-api-revoke-sub-15c6a7c6-8b65k\" (UID: \"3c0c479d-7754-4d33-bdac-72ff5f82900e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"
Nov 03 21:04:58.393332 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.393243    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"
Nov 03 21:04:58.671592 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:58.671562    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"]
Nov 03 21:04:58.673605 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:04:58.673579    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0c479d_7754_4d33_bdac_72ff5f82900e.slice/crio-77d7f56d7ece06439b365d3fb2d4fc7bb88bddd2a021afe62968518cde9aceb1 WatchSource:0}: Error finding container 77d7f56d7ece06439b365d3fb2d4fc7bb88bddd2a021afe62968518cde9aceb1: Status 404 returned error can't find the container with id 77d7f56d7ece06439b365d3fb2d4fc7bb88bddd2a021afe62968518cde9aceb1
Nov 03 21:04:59.194578 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:59.186400    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k" event={"ID":"3c0c479d-7754-4d33-bdac-72ff5f82900e","Type":"ContainerStarted","Data":"cb8e07ccd86120e53cdb16851a8cb804f5023f8ab2fe1a9fc656df2fc04056ff"}
Nov 03 21:04:59.194578 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:59.186493    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k" event={"ID":"3c0c479d-7754-4d33-bdac-72ff5f82900e","Type":"ContainerStarted","Data":"77d7f56d7ece06439b365d3fb2d4fc7bb88bddd2a021afe62968518cde9aceb1"}
Nov 03 21:04:59.214191 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:04:59.214128    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k" podStartSLOduration=1.214109505 podStartE2EDuration="1.214109505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:59.213707993 +0000 UTC m=+5251.368610413" watchObservedRunningTime="2026-11-03 21:04:59.214109505 +0000 UTC m=+5251.369011926"
Nov 03 21:05:00.193990 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:00.193953    5659 generic.go:356] "Generic (PLEG): container finished" podID="3c0c479d-7754-4d33-bdac-72ff5f82900e" containerID="cb8e07ccd86120e53cdb16851a8cb804f5023f8ab2fe1a9fc656df2fc04056ff" exitCode=0
Nov 03 21:05:00.194191 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:00.194035    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k" event={"ID":"3c0c479d-7754-4d33-bdac-72ff5f82900e","Type":"ContainerDied","Data":"cb8e07ccd86120e53cdb16851a8cb804f5023f8ab2fe1a9fc656df2fc04056ff"}
Nov 03 21:05:01.339030 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:01.339005    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"
Nov 03 21:05:01.433282 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:01.433249    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0c479d-7754-4d33-bdac-72ff5f82900e-kube-api-access-jhqcs\" (UniqueName: \"kubernetes.io/projected/3c0c479d-7754-4d33-bdac-72ff5f82900e-kube-api-access-jhqcs\") pod \"3c0c479d-7754-4d33-bdac-72ff5f82900e\" (UID: \"3c0c479d-7754-4d33-bdac-72ff5f82900e\") "
Nov 03 21:05:01.433461 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:01.433335    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-xks-ca\") pod \"3c0c479d-7754-4d33-bdac-72ff5f82900e\" (UID: \"3c0c479d-7754-4d33-bdac-72ff5f82900e\") "
Nov 03 21:05:01.433461 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:01.433404    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-openshift-ca\") pod \"3c0c479d-7754-4d33-bdac-72ff5f82900e\" (UID: \"3c0c479d-7754-4d33-bdac-72ff5f82900e\") "
Nov 03 21:05:01.433823 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:01.433798    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-openshift-ca" pod "3c0c479d-7754-4d33-bdac-72ff5f82900e" (UID: "3c0c479d-7754-4d33-bdac-72ff5f82900e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:01.435458 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:01.435430    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-xks-ca" pod "3c0c479d-7754-4d33-bdac-72ff5f82900e" (UID: "3c0c479d-7754-4d33-bdac-72ff5f82900e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:01.435572 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:01.435519    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0c479d-7754-4d33-bdac-72ff5f82900e-kube-api-access-jhqcs" pod "3c0c479d-7754-4d33-bdac-72ff5f82900e" (UID: "3c0c479d-7754-4d33-bdac-72ff5f82900e"). InnerVolumeSpecName "kube-api-access-jhqcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:01.534916 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:01.534880    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhqcs\" (UniqueName: \"kubernetes.io/projected/3c0c479d-7754-4d33-bdac-72ff5f82900e-kube-api-access-jhqcs\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:01.534916 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:01.534911    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:01.534916 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:01.534922    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0c479d-7754-4d33-bdac-72ff5f82900e-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:02.206985 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:02.206953    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k"
Nov 03 21:05:02.207176 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:02.206987    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15c6a7c6-8b65k" event={"ID":"3c0c479d-7754-4d33-bdac-72ff5f82900e","Type":"ContainerDied","Data":"77d7f56d7ece06439b365d3fb2d4fc7bb88bddd2a021afe62968518cde9aceb1"}
Nov 03 21:05:02.207176 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:02.207038    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77d7f56d7ece06439b365d3fb2d4fc7bb88bddd2a021afe62968518cde9aceb1"
Nov 03 21:05:07.938844 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:07.938806    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:08.255764 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.255733    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:05:08.287779 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.287746    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"]
Nov 03 21:05:08.288261 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.288245    5659 cpu_manager.go:414] "containerMap: removing container" podUID="3c0c479d-7754-4d33-bdac-72ff5f82900e" containerName="revoke-keys"
Nov 03 21:05:08.288314 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.288264    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0c479d-7754-4d33-bdac-72ff5f82900e" containerName="revoke-keys"
Nov 03 21:05:08.288370 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.288360    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0c479d-7754-4d33-bdac-72ff5f82900e" containerName="revoke-keys"
Nov 03 21:05:08.291695 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.291677    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"
Nov 03 21:05:08.297371 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.297344    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:05:08.298021 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.298002    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:05:08.301588 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.301563    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"]
Nov 03 21:05:08.310594 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.310567    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:05:08.310872 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.310849    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:05:08.400924 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.400889    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af841994-6b39-497e-a2ff-471199532c44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59406795-xsr68\" (UID: \"af841994-6b39-497e-a2ff-471199532c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"
Nov 03 21:05:08.400924 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.400926    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv67f\" (UniqueName: \"kubernetes.io/projected/af841994-6b39-497e-a2ff-471199532c44-kube-api-access-xv67f\") pod \"maas-api-revoke-sub-59406795-xsr68\" (UID: \"af841994-6b39-497e-a2ff-471199532c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"
Nov 03 21:05:08.401172 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.400951    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af841994-6b39-497e-a2ff-471199532c44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59406795-xsr68\" (UID: \"af841994-6b39-497e-a2ff-471199532c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"
Nov 03 21:05:08.502434 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.502390    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af841994-6b39-497e-a2ff-471199532c44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59406795-xsr68\" (UID: \"af841994-6b39-497e-a2ff-471199532c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"
Nov 03 21:05:08.502434 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.502430    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv67f\" (UniqueName: \"kubernetes.io/projected/af841994-6b39-497e-a2ff-471199532c44-kube-api-access-xv67f\") pod \"maas-api-revoke-sub-59406795-xsr68\" (UID: \"af841994-6b39-497e-a2ff-471199532c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"
Nov 03 21:05:08.502738 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.502455    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af841994-6b39-497e-a2ff-471199532c44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59406795-xsr68\" (UID: \"af841994-6b39-497e-a2ff-471199532c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"
Nov 03 21:05:08.503166 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.503134    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af841994-6b39-497e-a2ff-471199532c44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59406795-xsr68\" (UID: \"af841994-6b39-497e-a2ff-471199532c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"
Nov 03 21:05:08.504873 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.504852    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af841994-6b39-497e-a2ff-471199532c44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59406795-xsr68\" (UID: \"af841994-6b39-497e-a2ff-471199532c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"
Nov 03 21:05:08.515607 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.515553    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv67f\" (UniqueName: \"kubernetes.io/projected/af841994-6b39-497e-a2ff-471199532c44-kube-api-access-xv67f\") pod \"maas-api-revoke-sub-59406795-xsr68\" (UID: \"af841994-6b39-497e-a2ff-471199532c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"
Nov 03 21:05:08.607098 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.607058    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"
Nov 03 21:05:08.886905 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:08.886871    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"]
Nov 03 21:05:08.888098 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:05:08.888076    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf841994_6b39_497e_a2ff_471199532c44.slice/crio-0c46ff03c4d945b90ea033b045f25afc4267df0184146bde6608acdc375d5494 WatchSource:0}: Error finding container 0c46ff03c4d945b90ea033b045f25afc4267df0184146bde6608acdc375d5494: Status 404 returned error can't find the container with id 0c46ff03c4d945b90ea033b045f25afc4267df0184146bde6608acdc375d5494
Nov 03 21:05:09.249681 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:09.249648    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68" event={"ID":"af841994-6b39-497e-a2ff-471199532c44","Type":"ContainerStarted","Data":"85b74c08b2e33d5d0bd834941efa571046bd0237fe128af5f1a16c5c4121a04b"}
Nov 03 21:05:09.250073 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:09.249699    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68" event={"ID":"af841994-6b39-497e-a2ff-471199532c44","Type":"ContainerStarted","Data":"0c46ff03c4d945b90ea033b045f25afc4267df0184146bde6608acdc375d5494"}
Nov 03 21:05:09.269630 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:09.269579    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68" podStartSLOduration=1.269564975 podStartE2EDuration="1.269564975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:09.267888411 +0000 UTC m=+5261.422790832" watchObservedRunningTime="2026-11-03 21:05:09.269564975 +0000 UTC m=+5261.424467463"
Nov 03 21:05:10.259754 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:10.259711    5659 generic.go:356] "Generic (PLEG): container finished" podID="af841994-6b39-497e-a2ff-471199532c44" containerID="85b74c08b2e33d5d0bd834941efa571046bd0237fe128af5f1a16c5c4121a04b" exitCode=0
Nov 03 21:05:10.260296 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:10.259800    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68" event={"ID":"af841994-6b39-497e-a2ff-471199532c44","Type":"ContainerDied","Data":"85b74c08b2e33d5d0bd834941efa571046bd0237fe128af5f1a16c5c4121a04b"}
Nov 03 21:05:11.467858 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:11.467507    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"
Nov 03 21:05:11.539857 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:11.539770    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af841994-6b39-497e-a2ff-471199532c44-kube-api-access-xv67f\" (UniqueName: \"kubernetes.io/projected/af841994-6b39-497e-a2ff-471199532c44-kube-api-access-xv67f\") pod \"af841994-6b39-497e-a2ff-471199532c44\" (UID: \"af841994-6b39-497e-a2ff-471199532c44\") "
Nov 03 21:05:11.539857 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:11.539855    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af841994-6b39-497e-a2ff-471199532c44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af841994-6b39-497e-a2ff-471199532c44-maas-api-xks-ca\") pod \"af841994-6b39-497e-a2ff-471199532c44\" (UID: \"af841994-6b39-497e-a2ff-471199532c44\") "
Nov 03 21:05:11.540103 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:11.539927    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af841994-6b39-497e-a2ff-471199532c44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af841994-6b39-497e-a2ff-471199532c44-maas-api-openshift-ca\") pod \"af841994-6b39-497e-a2ff-471199532c44\" (UID: \"af841994-6b39-497e-a2ff-471199532c44\") "
Nov 03 21:05:11.540501 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:11.540457    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af841994-6b39-497e-a2ff-471199532c44-maas-api-openshift-ca" pod "af841994-6b39-497e-a2ff-471199532c44" (UID: "af841994-6b39-497e-a2ff-471199532c44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:11.542577 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:11.542549    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af841994-6b39-497e-a2ff-471199532c44-maas-api-xks-ca" pod "af841994-6b39-497e-a2ff-471199532c44" (UID: "af841994-6b39-497e-a2ff-471199532c44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:11.542763 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:11.542729    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af841994-6b39-497e-a2ff-471199532c44-kube-api-access-xv67f" pod "af841994-6b39-497e-a2ff-471199532c44" (UID: "af841994-6b39-497e-a2ff-471199532c44"). InnerVolumeSpecName "kube-api-access-xv67f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:11.641436 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:11.641378    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af841994-6b39-497e-a2ff-471199532c44-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:11.641436 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:11.641437    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af841994-6b39-497e-a2ff-471199532c44-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:11.641436 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:11.641455    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv67f\" (UniqueName: \"kubernetes.io/projected/af841994-6b39-497e-a2ff-471199532c44-kube-api-access-xv67f\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:12.277826 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:12.277728    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68" event={"ID":"af841994-6b39-497e-a2ff-471199532c44","Type":"ContainerDied","Data":"0c46ff03c4d945b90ea033b045f25afc4267df0184146bde6608acdc375d5494"}
Nov 03 21:05:12.277826 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:12.277792    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c46ff03c4d945b90ea033b045f25afc4267df0184146bde6608acdc375d5494"
Nov 03 21:05:12.277826 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:12.277749    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59406795-xsr68"
Nov 03 21:05:15.216060 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:05:15.216023    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:05:18.514256 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.514212    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"]
Nov 03 21:05:18.514892 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.514869    5659 cpu_manager.go:414] "containerMap: removing container" podUID="af841994-6b39-497e-a2ff-471199532c44" containerName="revoke-keys"
Nov 03 21:05:18.514980 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.514894    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af841994-6b39-497e-a2ff-471199532c44" containerName="revoke-keys"
Nov 03 21:05:18.515069 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.515055    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="af841994-6b39-497e-a2ff-471199532c44" containerName="revoke-keys"
Nov 03 21:05:18.518315 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.518293    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"
Nov 03 21:05:18.529665 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.529606    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:05:18.529665 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.529625    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:05:18.534169 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.534139    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"]
Nov 03 21:05:18.567341 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.567298    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:05:18.567543 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.567391    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:05:18.609869 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.609833    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-427c0937-q94bc\" (UID: \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"
Nov 03 21:05:18.610041 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.609981    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjv7n\" (UniqueName: \"kubernetes.io/projected/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-kube-api-access-pjv7n\") pod \"maas-api-revoke-sub-427c0937-q94bc\" (UID: \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"
Nov 03 21:05:18.610083 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.610064    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-427c0937-q94bc\" (UID: \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"
Nov 03 21:05:18.710960 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.710921    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-427c0937-q94bc\" (UID: \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"
Nov 03 21:05:18.711151 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.711106    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjv7n\" (UniqueName: \"kubernetes.io/projected/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-kube-api-access-pjv7n\") pod \"maas-api-revoke-sub-427c0937-q94bc\" (UID: \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"
Nov 03 21:05:18.711203 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.711169    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-427c0937-q94bc\" (UID: \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"
Nov 03 21:05:18.711697 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.711678    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-427c0937-q94bc\" (UID: \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"
Nov 03 21:05:18.713607 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.713584    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-427c0937-q94bc\" (UID: \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"
Nov 03 21:05:18.724075 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.724049    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjv7n\" (UniqueName: \"kubernetes.io/projected/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-kube-api-access-pjv7n\") pod \"maas-api-revoke-sub-427c0937-q94bc\" (UID: \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"
Nov 03 21:05:18.841977 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:18.841891    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"
Nov 03 21:05:19.120400 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:19.120371    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"]
Nov 03 21:05:19.121729 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:05:19.121694    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf8d5c7_5b84_412d_ac3f_f08000e22cb8.slice/crio-47755814d08ca4bcd879da54df3dede19bab11c3210a701a071500ab32edbe6b WatchSource:0}: Error finding container 47755814d08ca4bcd879da54df3dede19bab11c3210a701a071500ab32edbe6b: Status 404 returned error can't find the container with id 47755814d08ca4bcd879da54df3dede19bab11c3210a701a071500ab32edbe6b
Nov 03 21:05:19.318806 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:19.318770    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc" event={"ID":"abf8d5c7-5b84-412d-ac3f-f08000e22cb8","Type":"ContainerStarted","Data":"47755814d08ca4bcd879da54df3dede19bab11c3210a701a071500ab32edbe6b"}
Nov 03 21:05:20.328506 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:20.328393    5659 generic.go:356] "Generic (PLEG): container finished" podID="abf8d5c7-5b84-412d-ac3f-f08000e22cb8" containerID="aa1a62a19d8655c0c4869ebf73f0ef17f111bc2cade4f2a008c0654d27e1c3c9" exitCode=0
Nov 03 21:05:20.328506 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:20.328474    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc" event={"ID":"abf8d5c7-5b84-412d-ac3f-f08000e22cb8","Type":"ContainerDied","Data":"aa1a62a19d8655c0c4869ebf73f0ef17f111bc2cade4f2a008c0654d27e1c3c9"}
Nov 03 21:05:21.530061 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:21.530033    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"
Nov 03 21:05:21.563237 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:21.562331    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-openshift-ca\") pod \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\" (UID: \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\") "
Nov 03 21:05:21.563237 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:21.562451    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-kube-api-access-pjv7n\" (UniqueName: \"kubernetes.io/projected/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-kube-api-access-pjv7n\") pod \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\" (UID: \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\") "
Nov 03 21:05:21.563237 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:21.562552    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-xks-ca\") pod \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\" (UID: \"abf8d5c7-5b84-412d-ac3f-f08000e22cb8\") "
Nov 03 21:05:21.564252 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:21.563957    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-openshift-ca" pod "abf8d5c7-5b84-412d-ac3f-f08000e22cb8" (UID: "abf8d5c7-5b84-412d-ac3f-f08000e22cb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:21.600631 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:21.590366    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-kube-api-access-pjv7n" pod "abf8d5c7-5b84-412d-ac3f-f08000e22cb8" (UID: "abf8d5c7-5b84-412d-ac3f-f08000e22cb8"). InnerVolumeSpecName "kube-api-access-pjv7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:21.603657 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:21.603604    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-xks-ca" pod "abf8d5c7-5b84-412d-ac3f-f08000e22cb8" (UID: "abf8d5c7-5b84-412d-ac3f-f08000e22cb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:21.664518 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:21.664450    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:21.664796 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:21.664772    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjv7n\" (UniqueName: \"kubernetes.io/projected/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-kube-api-access-pjv7n\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:21.664918 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:21.664902    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf8d5c7-5b84-412d-ac3f-f08000e22cb8-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:22.344474 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:22.344375    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc" event={"ID":"abf8d5c7-5b84-412d-ac3f-f08000e22cb8","Type":"ContainerDied","Data":"47755814d08ca4bcd879da54df3dede19bab11c3210a701a071500ab32edbe6b"}
Nov 03 21:05:22.344474 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:22.344421    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-427c0937-q94bc"
Nov 03 21:05:22.344474 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:22.344427    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47755814d08ca4bcd879da54df3dede19bab11c3210a701a071500ab32edbe6b"
Nov 03 21:05:28.718943 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.718903    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"]
Nov 03 21:05:28.719396 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.719381    5659 cpu_manager.go:414] "containerMap: removing container" podUID="abf8d5c7-5b84-412d-ac3f-f08000e22cb8" containerName="revoke-keys"
Nov 03 21:05:28.719396 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.719396    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf8d5c7-5b84-412d-ac3f-f08000e22cb8" containerName="revoke-keys"
Nov 03 21:05:28.719507 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.719468    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf8d5c7-5b84-412d-ac3f-f08000e22cb8" containerName="revoke-keys"
Nov 03 21:05:28.722967 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.722943    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"
Nov 03 21:05:28.728100 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.728080    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:05:28.728976 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.728949    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:05:28.733253 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.733223    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"]
Nov 03 21:05:28.734755 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.734735    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:05:28.734867 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.734847    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:05:28.835036 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.834999    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc2c9986-xcw27\" (UID: \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"
Nov 03 21:05:28.835240 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.835148    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc2c9986-xcw27\" (UID: \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"
Nov 03 21:05:28.835297 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.835248    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxzp\" (UniqueName: \"kubernetes.io/projected/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-kube-api-access-qfxzp\") pod \"maas-api-revoke-sub-dc2c9986-xcw27\" (UID: \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"
Nov 03 21:05:28.935831 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.935791    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc2c9986-xcw27\" (UID: \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"
Nov 03 21:05:28.936020 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.935856    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxzp\" (UniqueName: \"kubernetes.io/projected/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-kube-api-access-qfxzp\") pod \"maas-api-revoke-sub-dc2c9986-xcw27\" (UID: \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"
Nov 03 21:05:28.936020 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.935899    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc2c9986-xcw27\" (UID: \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"
Nov 03 21:05:28.936563 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.936542    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc2c9986-xcw27\" (UID: \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"
Nov 03 21:05:28.938191 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.938166    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc2c9986-xcw27\" (UID: \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"
Nov 03 21:05:28.948583 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:28.948554    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxzp\" (UniqueName: \"kubernetes.io/projected/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-kube-api-access-qfxzp\") pod \"maas-api-revoke-sub-dc2c9986-xcw27\" (UID: \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"
Nov 03 21:05:29.040052 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:29.039975    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"
Nov 03 21:05:29.318610 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:29.318579    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"]
Nov 03 21:05:29.320095 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:05:29.320070    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d3b5b6_eec3_4402_9b5a_6257ff5f88b0.slice/crio-b34eed3dc299621e049057677e7aa1e08038259977a3e274c26156cf2d35c217 WatchSource:0}: Error finding container b34eed3dc299621e049057677e7aa1e08038259977a3e274c26156cf2d35c217: Status 404 returned error can't find the container with id b34eed3dc299621e049057677e7aa1e08038259977a3e274c26156cf2d35c217
Nov 03 21:05:29.386742 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:29.386703    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27" event={"ID":"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0","Type":"ContainerStarted","Data":"b34eed3dc299621e049057677e7aa1e08038259977a3e274c26156cf2d35c217"}
Nov 03 21:05:30.395572 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:30.395466    5659 generic.go:356] "Generic (PLEG): container finished" podID="01d3b5b6-eec3-4402-9b5a-6257ff5f88b0" containerID="a35c39b38d2b2d51cb904b7d0d40f1ca4f8e290ded3f227e6b6b5fcc388ce651" exitCode=0
Nov 03 21:05:30.395957 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:30.395562    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27" event={"ID":"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0","Type":"ContainerDied","Data":"a35c39b38d2b2d51cb904b7d0d40f1ca4f8e290ded3f227e6b6b5fcc388ce651"}
Nov 03 21:05:31.620149 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:31.620125    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"
Nov 03 21:05:31.673955 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:31.672844    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-kube-api-access-qfxzp\" (UniqueName: \"kubernetes.io/projected/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-kube-api-access-qfxzp\") pod \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\" (UID: \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\") "
Nov 03 21:05:31.673955 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:31.673048    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-openshift-ca\") pod \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\" (UID: \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\") "
Nov 03 21:05:31.673955 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:31.673107    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-xks-ca\") pod \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\" (UID: \"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0\") "
Nov 03 21:05:31.675629 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:31.675575    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-openshift-ca" pod "01d3b5b6-eec3-4402-9b5a-6257ff5f88b0" (UID: "01d3b5b6-eec3-4402-9b5a-6257ff5f88b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:31.692120 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:31.692066    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-xks-ca" pod "01d3b5b6-eec3-4402-9b5a-6257ff5f88b0" (UID: "01d3b5b6-eec3-4402-9b5a-6257ff5f88b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:31.695990 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:31.695963    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-kube-api-access-qfxzp" pod "01d3b5b6-eec3-4402-9b5a-6257ff5f88b0" (UID: "01d3b5b6-eec3-4402-9b5a-6257ff5f88b0"). InnerVolumeSpecName "kube-api-access-qfxzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:31.774138 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:31.774090    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:31.774138 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:31.774131    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:31.774138 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:31.774144    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfxzp\" (UniqueName: \"kubernetes.io/projected/01d3b5b6-eec3-4402-9b5a-6257ff5f88b0-kube-api-access-qfxzp\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.409985 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:32.409888    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27"
Nov 03 21:05:32.409985 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:32.409917    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc2c9986-xcw27" event={"ID":"01d3b5b6-eec3-4402-9b5a-6257ff5f88b0","Type":"ContainerDied","Data":"b34eed3dc299621e049057677e7aa1e08038259977a3e274c26156cf2d35c217"}
Nov 03 21:05:32.409985 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:32.409977    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b34eed3dc299621e049057677e7aa1e08038259977a3e274c26156cf2d35c217"
Nov 03 21:05:38.934453 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:38.934418    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"]
Nov 03 21:05:38.935649 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:38.935618    5659 cpu_manager.go:414] "containerMap: removing container" podUID="01d3b5b6-eec3-4402-9b5a-6257ff5f88b0" containerName="revoke-keys"
Nov 03 21:05:38.935649 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:38.935650    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d3b5b6-eec3-4402-9b5a-6257ff5f88b0" containerName="revoke-keys"
Nov 03 21:05:38.935995 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:38.935980    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d3b5b6-eec3-4402-9b5a-6257ff5f88b0" containerName="revoke-keys"
Nov 03 21:05:38.946212 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:38.946184    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"
Nov 03 21:05:38.949576 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:38.949549    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"]
Nov 03 21:05:38.952726 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:38.952699    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:05:38.952839 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:38.952768    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:05:38.980142 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:38.980108    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:05:38.980330 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:38.980176    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:05:39.055949 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:39.055903    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjblz\" (UniqueName: \"kubernetes.io/projected/c8f1dffc-25da-4765-a482-1ed6ba730579-kube-api-access-zjblz\") pod \"maas-api-revoke-sub-624ded16-m5x58\" (UID: \"c8f1dffc-25da-4765-a482-1ed6ba730579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"
Nov 03 21:05:39.056148 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:39.056039    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-624ded16-m5x58\" (UID: \"c8f1dffc-25da-4765-a482-1ed6ba730579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"
Nov 03 21:05:39.056197 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:39.056182    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-624ded16-m5x58\" (UID: \"c8f1dffc-25da-4765-a482-1ed6ba730579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"
Nov 03 21:05:39.157187 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:39.157143    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjblz\" (UniqueName: \"kubernetes.io/projected/c8f1dffc-25da-4765-a482-1ed6ba730579-kube-api-access-zjblz\") pod \"maas-api-revoke-sub-624ded16-m5x58\" (UID: \"c8f1dffc-25da-4765-a482-1ed6ba730579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"
Nov 03 21:05:39.157349 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:39.157193    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-624ded16-m5x58\" (UID: \"c8f1dffc-25da-4765-a482-1ed6ba730579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"
Nov 03 21:05:39.157349 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:39.157261    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-624ded16-m5x58\" (UID: \"c8f1dffc-25da-4765-a482-1ed6ba730579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"
Nov 03 21:05:39.157962 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:39.157932    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-624ded16-m5x58\" (UID: \"c8f1dffc-25da-4765-a482-1ed6ba730579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"
Nov 03 21:05:39.159612 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:39.159592    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-624ded16-m5x58\" (UID: \"c8f1dffc-25da-4765-a482-1ed6ba730579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"
Nov 03 21:05:39.169914 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:39.169884    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjblz\" (UniqueName: \"kubernetes.io/projected/c8f1dffc-25da-4765-a482-1ed6ba730579-kube-api-access-zjblz\") pod \"maas-api-revoke-sub-624ded16-m5x58\" (UID: \"c8f1dffc-25da-4765-a482-1ed6ba730579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"
Nov 03 21:05:39.263851 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:39.263818    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"
Nov 03 21:05:39.541729 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:39.541702    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"]
Nov 03 21:05:39.543525 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:05:39.543493    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f1dffc_25da_4765_a482_1ed6ba730579.slice/crio-171036eeed51c344a27c0764cc426c174fc9532dabca1ddc23a957315bd17fdf WatchSource:0}: Error finding container 171036eeed51c344a27c0764cc426c174fc9532dabca1ddc23a957315bd17fdf: Status 404 returned error can't find the container with id 171036eeed51c344a27c0764cc426c174fc9532dabca1ddc23a957315bd17fdf
Nov 03 21:05:40.461737 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:40.461705    5659 generic.go:356] "Generic (PLEG): container finished" podID="c8f1dffc-25da-4765-a482-1ed6ba730579" containerID="d9a49f2df35a0d8cb3097db6894a75078545593d43885e7b5b39a48e14af32dd" exitCode=0
Nov 03 21:05:40.462241 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:40.461788    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58" event={"ID":"c8f1dffc-25da-4765-a482-1ed6ba730579","Type":"ContainerDied","Data":"d9a49f2df35a0d8cb3097db6894a75078545593d43885e7b5b39a48e14af32dd"}
Nov 03 21:05:40.462241 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:40.461844    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58" event={"ID":"c8f1dffc-25da-4765-a482-1ed6ba730579","Type":"ContainerStarted","Data":"171036eeed51c344a27c0764cc426c174fc9532dabca1ddc23a957315bd17fdf"}
Nov 03 21:05:41.673456 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:41.673416    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"
Nov 03 21:05:41.787537 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:41.787406    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f1dffc-25da-4765-a482-1ed6ba730579-kube-api-access-zjblz\" (UniqueName: \"kubernetes.io/projected/c8f1dffc-25da-4765-a482-1ed6ba730579-kube-api-access-zjblz\") pod \"c8f1dffc-25da-4765-a482-1ed6ba730579\" (UID: \"c8f1dffc-25da-4765-a482-1ed6ba730579\") "
Nov 03 21:05:41.787935 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:41.787908    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-openshift-ca\") pod \"c8f1dffc-25da-4765-a482-1ed6ba730579\" (UID: \"c8f1dffc-25da-4765-a482-1ed6ba730579\") "
Nov 03 21:05:41.788688 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:41.788548    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-openshift-ca" pod "c8f1dffc-25da-4765-a482-1ed6ba730579" (UID: "c8f1dffc-25da-4765-a482-1ed6ba730579"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:41.789278 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:41.789246    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-xks-ca\") pod \"c8f1dffc-25da-4765-a482-1ed6ba730579\" (UID: \"c8f1dffc-25da-4765-a482-1ed6ba730579\") "
Nov 03 21:05:41.789850 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:41.789831    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:41.790369 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:41.790340    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f1dffc-25da-4765-a482-1ed6ba730579-kube-api-access-zjblz" pod "c8f1dffc-25da-4765-a482-1ed6ba730579" (UID: "c8f1dffc-25da-4765-a482-1ed6ba730579"). InnerVolumeSpecName "kube-api-access-zjblz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:41.799821 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:41.799795    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-xks-ca" pod "c8f1dffc-25da-4765-a482-1ed6ba730579" (UID: "c8f1dffc-25da-4765-a482-1ed6ba730579"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:41.891324 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:41.891250    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f1dffc-25da-4765-a482-1ed6ba730579-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:41.891324 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:41.891323    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjblz\" (UniqueName: \"kubernetes.io/projected/c8f1dffc-25da-4765-a482-1ed6ba730579-kube-api-access-zjblz\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:42.478316 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:42.478285    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58"
Nov 03 21:05:42.478316 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:42.478293    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624ded16-m5x58" event={"ID":"c8f1dffc-25da-4765-a482-1ed6ba730579","Type":"ContainerDied","Data":"171036eeed51c344a27c0764cc426c174fc9532dabca1ddc23a957315bd17fdf"}
Nov 03 21:05:42.478575 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:42.478337    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="171036eeed51c344a27c0764cc426c174fc9532dabca1ddc23a957315bd17fdf"
Nov 03 21:05:45.216783 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:05:45.216744    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:05:49.148756 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.148720    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"]
Nov 03 21:05:49.149377 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.149355    5659 cpu_manager.go:414] "containerMap: removing container" podUID="c8f1dffc-25da-4765-a482-1ed6ba730579" containerName="revoke-keys"
Nov 03 21:05:49.149473 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.149379    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f1dffc-25da-4765-a482-1ed6ba730579" containerName="revoke-keys"
Nov 03 21:05:49.149570 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.149554    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f1dffc-25da-4765-a482-1ed6ba730579" containerName="revoke-keys"
Nov 03 21:05:49.152901 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.152879    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"
Nov 03 21:05:49.158404 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.158364    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:05:49.158404 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.158370    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:05:49.158627 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.158378    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:05:49.158627 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.158563    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:05:49.162959 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.162936    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"]
Nov 03 21:05:49.265530 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.265461    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78hnn\" (UniqueName: \"kubernetes.io/projected/5f271b32-dfec-4810-8c32-d6511aec26b0-kube-api-access-78hnn\") pod \"maas-api-revoke-sub-846860d7-qcbt5\" (UID: \"5f271b32-dfec-4810-8c32-d6511aec26b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"
Nov 03 21:05:49.265717 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.265538    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-846860d7-qcbt5\" (UID: \"5f271b32-dfec-4810-8c32-d6511aec26b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"
Nov 03 21:05:49.265717 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.265654    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-846860d7-qcbt5\" (UID: \"5f271b32-dfec-4810-8c32-d6511aec26b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"
Nov 03 21:05:49.366942 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.366907    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hnn\" (UniqueName: \"kubernetes.io/projected/5f271b32-dfec-4810-8c32-d6511aec26b0-kube-api-access-78hnn\") pod \"maas-api-revoke-sub-846860d7-qcbt5\" (UID: \"5f271b32-dfec-4810-8c32-d6511aec26b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"
Nov 03 21:05:49.367115 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.366949    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-846860d7-qcbt5\" (UID: \"5f271b32-dfec-4810-8c32-d6511aec26b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"
Nov 03 21:05:49.367115 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.367012    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-846860d7-qcbt5\" (UID: \"5f271b32-dfec-4810-8c32-d6511aec26b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"
Nov 03 21:05:49.367816 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.367786    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-846860d7-qcbt5\" (UID: \"5f271b32-dfec-4810-8c32-d6511aec26b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"
Nov 03 21:05:49.369497 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.369458    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-846860d7-qcbt5\" (UID: \"5f271b32-dfec-4810-8c32-d6511aec26b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"
Nov 03 21:05:49.379304 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.379282    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78hnn\" (UniqueName: \"kubernetes.io/projected/5f271b32-dfec-4810-8c32-d6511aec26b0-kube-api-access-78hnn\") pod \"maas-api-revoke-sub-846860d7-qcbt5\" (UID: \"5f271b32-dfec-4810-8c32-d6511aec26b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"
Nov 03 21:05:49.469846 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.469813    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"
Nov 03 21:05:49.747335 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:49.747291    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"]
Nov 03 21:05:49.749212 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:05:49.749181    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f271b32_dfec_4810_8c32_d6511aec26b0.slice/crio-f4d0fc2f0e2254072817d45893deb7e4b1a0c80a84028bc13daebc6bb073f265 WatchSource:0}: Error finding container f4d0fc2f0e2254072817d45893deb7e4b1a0c80a84028bc13daebc6bb073f265: Status 404 returned error can't find the container with id f4d0fc2f0e2254072817d45893deb7e4b1a0c80a84028bc13daebc6bb073f265
Nov 03 21:05:50.527955 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:50.527912    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5" event={"ID":"5f271b32-dfec-4810-8c32-d6511aec26b0","Type":"ContainerStarted","Data":"36993f62214cc54d3c3dff3c30920fdb8ba2df6e284cf34cb2512fce153df715"}
Nov 03 21:05:50.528416 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:50.527980    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5" event={"ID":"5f271b32-dfec-4810-8c32-d6511aec26b0","Type":"ContainerStarted","Data":"f4d0fc2f0e2254072817d45893deb7e4b1a0c80a84028bc13daebc6bb073f265"}
Nov 03 21:05:50.547350 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:50.547295    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5" podStartSLOduration=1.5472822320000001 podStartE2EDuration="1.547282232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:50.546888266 +0000 UTC m=+5302.701790685" watchObservedRunningTime="2026-11-03 21:05:50.547282232 +0000 UTC m=+5302.702184645"
Nov 03 21:05:51.538578 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:51.538536    5659 generic.go:356] "Generic (PLEG): container finished" podID="5f271b32-dfec-4810-8c32-d6511aec26b0" containerID="36993f62214cc54d3c3dff3c30920fdb8ba2df6e284cf34cb2512fce153df715" exitCode=0
Nov 03 21:05:51.539136 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:51.538724    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5" event={"ID":"5f271b32-dfec-4810-8c32-d6511aec26b0","Type":"ContainerDied","Data":"36993f62214cc54d3c3dff3c30920fdb8ba2df6e284cf34cb2512fce153df715"}
Nov 03 21:05:52.707940 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:52.707911    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"
Nov 03 21:05:52.808337 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:52.808241    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-xks-ca\") pod \"5f271b32-dfec-4810-8c32-d6511aec26b0\" (UID: \"5f271b32-dfec-4810-8c32-d6511aec26b0\") "
Nov 03 21:05:52.808337 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:52.808311    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f271b32-dfec-4810-8c32-d6511aec26b0-kube-api-access-78hnn\" (UniqueName: \"kubernetes.io/projected/5f271b32-dfec-4810-8c32-d6511aec26b0-kube-api-access-78hnn\") pod \"5f271b32-dfec-4810-8c32-d6511aec26b0\" (UID: \"5f271b32-dfec-4810-8c32-d6511aec26b0\") "
Nov 03 21:05:52.808636 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:52.808346    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-openshift-ca\") pod \"5f271b32-dfec-4810-8c32-d6511aec26b0\" (UID: \"5f271b32-dfec-4810-8c32-d6511aec26b0\") "
Nov 03 21:05:52.808787 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:52.808761    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-openshift-ca" pod "5f271b32-dfec-4810-8c32-d6511aec26b0" (UID: "5f271b32-dfec-4810-8c32-d6511aec26b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:52.810511 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:52.810468    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-xks-ca" pod "5f271b32-dfec-4810-8c32-d6511aec26b0" (UID: "5f271b32-dfec-4810-8c32-d6511aec26b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:52.810626 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:52.810522    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f271b32-dfec-4810-8c32-d6511aec26b0-kube-api-access-78hnn" pod "5f271b32-dfec-4810-8c32-d6511aec26b0" (UID: "5f271b32-dfec-4810-8c32-d6511aec26b0"). InnerVolumeSpecName "kube-api-access-78hnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:52.910031 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:52.909992    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:52.910031 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:52.910025    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78hnn\" (UniqueName: \"kubernetes.io/projected/5f271b32-dfec-4810-8c32-d6511aec26b0-kube-api-access-78hnn\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:52.910031 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:52.910037    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f271b32-dfec-4810-8c32-d6511aec26b0-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:53.554787 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:53.554752    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5"
Nov 03 21:05:53.555006 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:53.554752    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-846860d7-qcbt5" event={"ID":"5f271b32-dfec-4810-8c32-d6511aec26b0","Type":"ContainerDied","Data":"f4d0fc2f0e2254072817d45893deb7e4b1a0c80a84028bc13daebc6bb073f265"}
Nov 03 21:05:53.555006 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:53.554879    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4d0fc2f0e2254072817d45893deb7e4b1a0c80a84028bc13daebc6bb073f265"
Nov 03 21:05:59.370546 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.370509    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"]
Nov 03 21:05:59.370990 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.370974    5659 cpu_manager.go:414] "containerMap: removing container" podUID="5f271b32-dfec-4810-8c32-d6511aec26b0" containerName="revoke-keys"
Nov 03 21:05:59.371050 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.370992    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f271b32-dfec-4810-8c32-d6511aec26b0" containerName="revoke-keys"
Nov 03 21:05:59.371182 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.371167    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f271b32-dfec-4810-8c32-d6511aec26b0" containerName="revoke-keys"
Nov 03 21:05:59.374206 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.374189    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"
Nov 03 21:05:59.381786 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.381714    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:05:59.381786 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.381782    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:05:59.381952 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.381869    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:05:59.381952 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.381927    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:05:59.385724 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.385699    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"]
Nov 03 21:05:59.478211 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.478164    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-808e10db-5j58r\" (UID: \"f311e211-3a06-48da-add0-3c5538aced9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"
Nov 03 21:05:59.478425 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.478207    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-808e10db-5j58r\" (UID: \"f311e211-3a06-48da-add0-3c5538aced9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"
Nov 03 21:05:59.478425 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.478392    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8t9n\" (UniqueName: \"kubernetes.io/projected/f311e211-3a06-48da-add0-3c5538aced9e-kube-api-access-l8t9n\") pod \"maas-api-revoke-sub-808e10db-5j58r\" (UID: \"f311e211-3a06-48da-add0-3c5538aced9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"
Nov 03 21:05:59.579898 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.579860    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8t9n\" (UniqueName: \"kubernetes.io/projected/f311e211-3a06-48da-add0-3c5538aced9e-kube-api-access-l8t9n\") pod \"maas-api-revoke-sub-808e10db-5j58r\" (UID: \"f311e211-3a06-48da-add0-3c5538aced9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"
Nov 03 21:05:59.580107 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.579928    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-808e10db-5j58r\" (UID: \"f311e211-3a06-48da-add0-3c5538aced9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"
Nov 03 21:05:59.580107 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.579951    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-808e10db-5j58r\" (UID: \"f311e211-3a06-48da-add0-3c5538aced9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"
Nov 03 21:05:59.580615 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.580591    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-808e10db-5j58r\" (UID: \"f311e211-3a06-48da-add0-3c5538aced9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"
Nov 03 21:05:59.582369 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.582349    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-808e10db-5j58r\" (UID: \"f311e211-3a06-48da-add0-3c5538aced9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"
Nov 03 21:05:59.592259 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.592232    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8t9n\" (UniqueName: \"kubernetes.io/projected/f311e211-3a06-48da-add0-3c5538aced9e-kube-api-access-l8t9n\") pod \"maas-api-revoke-sub-808e10db-5j58r\" (UID: \"f311e211-3a06-48da-add0-3c5538aced9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"
Nov 03 21:05:59.691510 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.691400    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"
Nov 03 21:05:59.972174 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:05:59.972145    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"]
Nov 03 21:05:59.973520 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:05:59.973491    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf311e211_3a06_48da_add0_3c5538aced9e.slice/crio-065ebb9021d7b13528d572bff12778d39108ab2655574ab735020d824af8e449 WatchSource:0}: Error finding container 065ebb9021d7b13528d572bff12778d39108ab2655574ab735020d824af8e449: Status 404 returned error can't find the container with id 065ebb9021d7b13528d572bff12778d39108ab2655574ab735020d824af8e449
Nov 03 21:06:00.598272 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:00.598183    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r" event={"ID":"f311e211-3a06-48da-add0-3c5538aced9e","Type":"ContainerStarted","Data":"7951e057f0af172349055b003831055d0ca3c58f083e81401c668da2388c3b6e"}
Nov 03 21:06:00.598272 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:00.598242    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r" event={"ID":"f311e211-3a06-48da-add0-3c5538aced9e","Type":"ContainerStarted","Data":"065ebb9021d7b13528d572bff12778d39108ab2655574ab735020d824af8e449"}
Nov 03 21:06:00.619939 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:00.619879    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r" podStartSLOduration=1.619863714 podStartE2EDuration="1.619863714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:00.617180759 +0000 UTC m=+5312.772083181" watchObservedRunningTime="2026-11-03 21:06:00.619863714 +0000 UTC m=+5312.774766133"
Nov 03 21:06:01.607688 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:01.607653    5659 generic.go:356] "Generic (PLEG): container finished" podID="f311e211-3a06-48da-add0-3c5538aced9e" containerID="7951e057f0af172349055b003831055d0ca3c58f083e81401c668da2388c3b6e" exitCode=0
Nov 03 21:06:01.608126 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:01.607742    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r" event={"ID":"f311e211-3a06-48da-add0-3c5538aced9e","Type":"ContainerDied","Data":"7951e057f0af172349055b003831055d0ca3c58f083e81401c668da2388c3b6e"}
Nov 03 21:06:02.804419 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:02.804393    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"
Nov 03 21:06:02.915836 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:02.915747    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-openshift-ca\") pod \"f311e211-3a06-48da-add0-3c5538aced9e\" (UID: \"f311e211-3a06-48da-add0-3c5538aced9e\") "
Nov 03 21:06:02.915836 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:02.915786    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f311e211-3a06-48da-add0-3c5538aced9e-kube-api-access-l8t9n\" (UniqueName: \"kubernetes.io/projected/f311e211-3a06-48da-add0-3c5538aced9e-kube-api-access-l8t9n\") pod \"f311e211-3a06-48da-add0-3c5538aced9e\" (UID: \"f311e211-3a06-48da-add0-3c5538aced9e\") "
Nov 03 21:06:02.916099 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:02.915980    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-xks-ca\") pod \"f311e211-3a06-48da-add0-3c5538aced9e\" (UID: \"f311e211-3a06-48da-add0-3c5538aced9e\") "
Nov 03 21:06:02.916204 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:02.916177    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-openshift-ca" pod "f311e211-3a06-48da-add0-3c5538aced9e" (UID: "f311e211-3a06-48da-add0-3c5538aced9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:02.916367 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:02.916349    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:02.918058 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:02.918036    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f311e211-3a06-48da-add0-3c5538aced9e-kube-api-access-l8t9n" pod "f311e211-3a06-48da-add0-3c5538aced9e" (UID: "f311e211-3a06-48da-add0-3c5538aced9e"). InnerVolumeSpecName "kube-api-access-l8t9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:02.918154 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:02.918073    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-xks-ca" pod "f311e211-3a06-48da-add0-3c5538aced9e" (UID: "f311e211-3a06-48da-add0-3c5538aced9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:03.017212 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:03.017177    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f311e211-3a06-48da-add0-3c5538aced9e-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:03.017212 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:03.017206    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8t9n\" (UniqueName: \"kubernetes.io/projected/f311e211-3a06-48da-add0-3c5538aced9e-kube-api-access-l8t9n\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:03.624329 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:03.624297    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r"
Nov 03 21:06:03.624503 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:03.624336    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808e10db-5j58r" event={"ID":"f311e211-3a06-48da-add0-3c5538aced9e","Type":"ContainerDied","Data":"065ebb9021d7b13528d572bff12778d39108ab2655574ab735020d824af8e449"}
Nov 03 21:06:03.624503 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:03.624391    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="065ebb9021d7b13528d572bff12778d39108ab2655574ab735020d824af8e449"
Nov 03 21:06:07.988650 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:07.988613    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:08.300156 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:08.300072    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:06:09.590452 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.590363    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"]
Nov 03 21:06:09.591039 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.591017    5659 cpu_manager.go:414] "containerMap: removing container" podUID="f311e211-3a06-48da-add0-3c5538aced9e" containerName="revoke-keys"
Nov 03 21:06:09.591125 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.591042    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f311e211-3a06-48da-add0-3c5538aced9e" containerName="revoke-keys"
Nov 03 21:06:09.591216 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.591202    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="f311e211-3a06-48da-add0-3c5538aced9e" containerName="revoke-keys"
Nov 03 21:06:09.594693 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.594671    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"
Nov 03 21:06:09.600496 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.600457    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:06:09.600652 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.600546    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:06:09.600851 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.600834    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:06:09.601002 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.600988    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:06:09.605122 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.605099    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"]
Nov 03 21:06:09.683949 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.683917    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlhx6\" (UniqueName: \"kubernetes.io/projected/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-kube-api-access-zlhx6\") pod \"maas-api-revoke-sub-f88012c8-qblm2\" (UID: \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"
Nov 03 21:06:09.684130 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.684000    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f88012c8-qblm2\" (UID: \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"
Nov 03 21:06:09.684130 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.684073    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f88012c8-qblm2\" (UID: \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"
Nov 03 21:06:09.785105 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.785065    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlhx6\" (UniqueName: \"kubernetes.io/projected/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-kube-api-access-zlhx6\") pod \"maas-api-revoke-sub-f88012c8-qblm2\" (UID: \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"
Nov 03 21:06:09.785302 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.785159    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f88012c8-qblm2\" (UID: \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"
Nov 03 21:06:09.785302 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.785181    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f88012c8-qblm2\" (UID: \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"
Nov 03 21:06:09.785840 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.785812    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f88012c8-qblm2\" (UID: \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"
Nov 03 21:06:09.787518 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.787475    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f88012c8-qblm2\" (UID: \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"
Nov 03 21:06:09.798191 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.798171    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlhx6\" (UniqueName: \"kubernetes.io/projected/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-kube-api-access-zlhx6\") pod \"maas-api-revoke-sub-f88012c8-qblm2\" (UID: \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"
Nov 03 21:06:09.910579 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:09.910519    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"
Nov 03 21:06:10.192161 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:10.192126    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"]
Nov 03 21:06:10.193875 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:06:10.193843    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd4dc3c_d76c_4c43_83a9_e0c06833e2a2.slice/crio-5fe2acede94d4088dceb2bc763aee3cea944b1de9b53fe963463ed17a79e7ca6 WatchSource:0}: Error finding container 5fe2acede94d4088dceb2bc763aee3cea944b1de9b53fe963463ed17a79e7ca6: Status 404 returned error can't find the container with id 5fe2acede94d4088dceb2bc763aee3cea944b1de9b53fe963463ed17a79e7ca6
Nov 03 21:06:10.669755 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:10.669708    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2" event={"ID":"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2","Type":"ContainerStarted","Data":"3e3fe79e9fe0370f59c3fe838690be99525966a2ce7fc24279c6e7f29a32f8d9"}
Nov 03 21:06:10.670180 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:10.669782    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2" event={"ID":"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2","Type":"ContainerStarted","Data":"5fe2acede94d4088dceb2bc763aee3cea944b1de9b53fe963463ed17a79e7ca6"}
Nov 03 21:06:10.691492 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:10.691433    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2" podStartSLOduration=1.691419773 podStartE2EDuration="1.691419773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:10.687894698 +0000 UTC m=+5322.842797118" watchObservedRunningTime="2026-11-03 21:06:10.691419773 +0000 UTC m=+5322.846322193"
Nov 03 21:06:11.684609 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:11.684563    5659 generic.go:356] "Generic (PLEG): container finished" podID="ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2" containerID="3e3fe79e9fe0370f59c3fe838690be99525966a2ce7fc24279c6e7f29a32f8d9" exitCode=0
Nov 03 21:06:11.685128 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:11.684666    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2" event={"ID":"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2","Type":"ContainerDied","Data":"3e3fe79e9fe0370f59c3fe838690be99525966a2ce7fc24279c6e7f29a32f8d9"}
Nov 03 21:06:12.872659 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:12.872625    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"
Nov 03 21:06:12.918179 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:12.918129    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-openshift-ca\") pod \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\" (UID: \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\") "
Nov 03 21:06:12.918358 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:12.918234    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-kube-api-access-zlhx6\" (UniqueName: \"kubernetes.io/projected/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-kube-api-access-zlhx6\") pod \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\" (UID: \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\") "
Nov 03 21:06:12.918358 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:12.918266    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-xks-ca\") pod \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\" (UID: \"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2\") "
Nov 03 21:06:12.919277 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:12.919243    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-openshift-ca" pod "ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2" (UID: "ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:12.923644 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:12.923604    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-kube-api-access-zlhx6" pod "ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2" (UID: "ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2"). InnerVolumeSpecName "kube-api-access-zlhx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:12.925929 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:12.925894    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-xks-ca" pod "ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2" (UID: "ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:13.019352 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:13.019313    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlhx6\" (UniqueName: \"kubernetes.io/projected/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-kube-api-access-zlhx6\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:13.019352 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:13.019346    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:13.019601 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:13.019360    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:13.701920 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:13.701887    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2"
Nov 03 21:06:13.702099 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:13.701912    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f88012c8-qblm2" event={"ID":"ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2","Type":"ContainerDied","Data":"5fe2acede94d4088dceb2bc763aee3cea944b1de9b53fe963463ed17a79e7ca6"}
Nov 03 21:06:13.702099 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:13.701972    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fe2acede94d4088dceb2bc763aee3cea944b1de9b53fe963463ed17a79e7ca6"
Nov 03 21:06:15.216525 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:06:15.216474    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:06:19.806673 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.806640    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"]
Nov 03 21:06:19.807153 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.807136    5659 cpu_manager.go:414] "containerMap: removing container" podUID="ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2" containerName="revoke-keys"
Nov 03 21:06:19.807216 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.807156    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2" containerName="revoke-keys"
Nov 03 21:06:19.807264 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.807234    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2" containerName="revoke-keys"
Nov 03 21:06:19.810676 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.810657    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"
Nov 03 21:06:19.815234 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.815101    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:06:19.815234 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.815101    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:06:19.816172 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.816152    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 21:06:19.816306 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.816209    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:06:19.820833 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.820809    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"]
Nov 03 21:06:19.886116 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.886076    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w27v\" (UniqueName: \"kubernetes.io/projected/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-kube-api-access-4w27v\") pod \"maas-api-revoke-sub-9cecac37-86x7c\" (UID: \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"
Nov 03 21:06:19.886286 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.886228    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cecac37-86x7c\" (UID: \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"
Nov 03 21:06:19.886286 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.886267    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cecac37-86x7c\" (UID: \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"
Nov 03 21:06:19.987072 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.987037    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cecac37-86x7c\" (UID: \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"
Nov 03 21:06:19.987255 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.987100    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w27v\" (UniqueName: \"kubernetes.io/projected/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-kube-api-access-4w27v\") pod \"maas-api-revoke-sub-9cecac37-86x7c\" (UID: \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"
Nov 03 21:06:19.987255 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.987171    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cecac37-86x7c\" (UID: \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"
Nov 03 21:06:19.989036 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.989003    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cecac37-86x7c\" (UID: \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"
Nov 03 21:06:19.990696 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:19.990666    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cecac37-86x7c\" (UID: \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"
Nov 03 21:06:20.000547 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:20.000519    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w27v\" (UniqueName: \"kubernetes.io/projected/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-kube-api-access-4w27v\") pod \"maas-api-revoke-sub-9cecac37-86x7c\" (UID: \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"
Nov 03 21:06:20.126680 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:20.126601    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"
Nov 03 21:06:20.403799 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:20.403769    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"]
Nov 03 21:06:20.404978 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:06:20.404955    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3cc375_dadb_4c72_b71a_4ec05fec5a8d.slice/crio-f8cb202559fc0769993926c120fc7ba56b29416546902eef4d06483dd6bfb532 WatchSource:0}: Error finding container f8cb202559fc0769993926c120fc7ba56b29416546902eef4d06483dd6bfb532: Status 404 returned error can't find the container with id f8cb202559fc0769993926c120fc7ba56b29416546902eef4d06483dd6bfb532
Nov 03 21:06:20.746690 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:20.746662    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c" event={"ID":"da3cc375-dadb-4c72-b71a-4ec05fec5a8d","Type":"ContainerStarted","Data":"f8cb202559fc0769993926c120fc7ba56b29416546902eef4d06483dd6bfb532"}
Nov 03 21:06:21.759302 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:21.759248    5659 generic.go:356] "Generic (PLEG): container finished" podID="da3cc375-dadb-4c72-b71a-4ec05fec5a8d" containerID="e85e0ec245a48e93fcf4d025dc07dfebe04d07aeb5215b03a9b0ce6d02c55a05" exitCode=0
Nov 03 21:06:21.760202 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:21.759426    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c" event={"ID":"da3cc375-dadb-4c72-b71a-4ec05fec5a8d","Type":"ContainerDied","Data":"e85e0ec245a48e93fcf4d025dc07dfebe04d07aeb5215b03a9b0ce6d02c55a05"}
Nov 03 21:06:22.914946 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:22.914918    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"
Nov 03 21:06:23.019926 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:23.019845    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-kube-api-access-4w27v\" (UniqueName: \"kubernetes.io/projected/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-kube-api-access-4w27v\") pod \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\" (UID: \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\") "
Nov 03 21:06:23.019926 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:23.019900    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-xks-ca\") pod \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\" (UID: \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\") "
Nov 03 21:06:23.020129 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:23.019972    5659 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-openshift-ca\") pod \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\" (UID: \"da3cc375-dadb-4c72-b71a-4ec05fec5a8d\") "
Nov 03 21:06:23.020376 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:23.020344    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-openshift-ca" pod "da3cc375-dadb-4c72-b71a-4ec05fec5a8d" (UID: "da3cc375-dadb-4c72-b71a-4ec05fec5a8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:23.021976 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:23.021950    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-xks-ca" pod "da3cc375-dadb-4c72-b71a-4ec05fec5a8d" (UID: "da3cc375-dadb-4c72-b71a-4ec05fec5a8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:23.022073 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:23.021997    5659 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-kube-api-access-4w27v" pod "da3cc375-dadb-4c72-b71a-4ec05fec5a8d" (UID: "da3cc375-dadb-4c72-b71a-4ec05fec5a8d"). InnerVolumeSpecName "kube-api-access-4w27v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:23.121928 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:23.121899    5659 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w27v\" (UniqueName: \"kubernetes.io/projected/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-kube-api-access-4w27v\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:23.121928 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:23.121925    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-xks-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:23.121928 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:23.121936    5659 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3cc375-dadb-4c72-b71a-4ec05fec5a8d-maas-api-openshift-ca\") on node \"ip-10-0-133-249.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:23.774972 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:23.774939    5659 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c"
Nov 03 21:06:23.775150 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:23.774975    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecac37-86x7c" event={"ID":"da3cc375-dadb-4c72-b71a-4ec05fec5a8d","Type":"ContainerDied","Data":"f8cb202559fc0769993926c120fc7ba56b29416546902eef4d06483dd6bfb532"}
Nov 03 21:06:23.775150 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:23.775027    5659 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8cb202559fc0769993926c120fc7ba56b29416546902eef4d06483dd6bfb532"
Nov 03 21:06:45.217318 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:06:45.217278    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:06:49.108896 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:06:49.108865    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66b59d468d-j8ndl_19730c58-ac26-42dc-bd3a-8ec0f6076264/authorino/0.log"
Nov 03 21:07:03.503904 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:03.503864    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:03.637710 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:03.637672    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:07.022717 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:07.022677    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"]
Nov 03 21:07:07.025594 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:07.025563    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a029e6c-qdmz4"]
Nov 03 21:07:07.968238 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:07.968206    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a" path="/var/lib/kubelet/pods/121b8fc8-36c2-4e69-ac1e-1fffe3d0e72a/volumes"
Nov 03 21:07:08.049680 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:08.049647    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:08.355707 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:08.355622    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:07:15.217042 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:07:15.217000    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:07:27.024665 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:27.024619    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"]
Nov 03 21:07:27.029660 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:27.029627    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901ba1c0-dptzp"]
Nov 03 21:07:27.968444 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:27.968405    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20531b22-e37e-437f-b65f-3afc0560b35b" path="/var/lib/kubelet/pods/20531b22-e37e-437f-b65f-3afc0560b35b/volumes"
Nov 03 21:07:41.538110 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:41.538016    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fxxs9_aa079182-95a7-4454-be04-45f06a41088f/manager/0.log"
Nov 03 21:07:41.748346 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:41.748300    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qkjdd_950e6c44-5e74-4c6c-9c38-0712a81af85a/manager/0.log"
Nov 03 21:07:42.261917 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:42.261889    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-76gw5_0f42d32f-c8f1-48b4-807a-76636151cb72/server/0.log"
Nov 03 21:07:42.683543 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:42.683437    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-564sz_67be1eda-e12d-4a81-9785-5c5c8c3989d2/proxy/0.log"
Nov 03 21:07:43.979556 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:43.979530    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1087b164-jchw2_508f00aa-c3ac-436a-a618-8ec2ec8362ed/revoke-keys/0.log"
Nov 03 21:07:44.159696 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:44.159666    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-15c6a7c6-8b65k_3c0c479d-7754-4d33-bdac-72ff5f82900e/revoke-keys/0.log"
Nov 03 21:07:44.341428 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:44.341309    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2bce983d-sbgcs_9eb5dbb1-5893-4c22-a76e-ea24fda0b78b/revoke-keys/0.log"
Nov 03 21:07:44.522123 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:44.522088    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-427c0937-q94bc_abf8d5c7-5b84-412d-ac3f-f08000e22cb8/revoke-keys/0.log"
Nov 03 21:07:44.703277 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:44.703193    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-59406795-xsr68_af841994-6b39-497e-a2ff-471199532c44/revoke-keys/0.log"
Nov 03 21:07:44.891463 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:44.891435    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-624ded16-m5x58_c8f1dffc-25da-4765-a482-1ed6ba730579/revoke-keys/0.log"
Nov 03 21:07:45.071415 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:45.071379    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-73d6ee39-hc2dv_72714747-8a61-43b2-b2d3-01e5b7157150/revoke-keys/0.log"
Nov 03 21:07:45.216898 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:07:45.216863    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:07:45.243359 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:45.243324    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7a27a9d3-fjltl_e8024e4b-e3af-4bc5-935f-dddc089e0ed1/revoke-keys/0.log"
Nov 03 21:07:45.433297 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:45.433225    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7a82f982-m2w9g_74fa04c3-5af3-4f34-bb22-2832cee2accc/revoke-keys/0.log"
Nov 03 21:07:45.610832 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:45.610802    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-808e10db-5j58r_f311e211-3a06-48da-add0-3c5538aced9e/revoke-keys/0.log"
Nov 03 21:07:45.780662 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:45.780626    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-846860d7-qcbt5_5f271b32-dfec-4810-8c32-d6511aec26b0/revoke-keys/0.log"
Nov 03 21:07:45.961726 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:45.961692    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-89487c84-fsl55_95e6bf6c-083d-4286-bef8-5cb3d637cef0/revoke-keys/0.log"
Nov 03 21:07:46.135612 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:46.135524    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9cecac37-86x7c_da3cc375-dadb-4c72-b71a-4ec05fec5a8d/revoke-keys/0.log"
Nov 03 21:07:46.305889 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:46.305859    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c9a412c5-4kf6m_72553983-bbaf-4120-894a-9187ef86eed8/revoke-keys/0.log"
Nov 03 21:07:46.480070 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:46.480039    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dc2c9986-xcw27_01d3b5b6-eec3-4402-9b5a-6257ff5f88b0/revoke-keys/0.log"
Nov 03 21:07:46.660654 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:46.660619    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e3d9a142-5z7td_bf494daf-114e-404a-8331-5820955a0c3f/revoke-keys/0.log"
Nov 03 21:07:46.832678 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:46.832600    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f88012c8-qblm2_ecd4dc3c-d76c-4c43-83a9-e0c06833e2a2/revoke-keys/0.log"
Nov 03 21:07:47.030323 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:47.030282    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"]
Nov 03 21:07:47.032923 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:47.032900    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89487c84-fsl55"]
Nov 03 21:07:47.968189 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:47.968153    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e6bf6c-083d-4286-bef8-5cb3d637cef0" path="/var/lib/kubelet/pods/95e6bf6c-083d-4286-bef8-5cb3d637cef0/volumes"
Nov 03 21:07:48.910497 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:48.910453    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66b59d468d-j8ndl_19730c58-ac26-42dc-bd3a-8ec0f6076264/authorino/0.log"
Nov 03 21:07:49.323936 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:49.323909    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pmwn5_a8606f49-1dc0-45c2-9720-be9731a089c4/manager/0.log"
Nov 03 21:07:49.503640 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:49.503610    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-t4sb5_856603f3-24e3-4d9e-b6dc-68dc3f57d18a/kuadrant-console-plugin/0.log"
Nov 03 21:07:51.353194 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:51.353162    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8qktw_fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a/discovery/0.log"
Nov 03 21:07:51.734687 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:51.734635    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79ddfb4884-vtwjp_727ca680-846f-4226-b4ee-1dad5f785224/kube-auth-proxy/0.log"
Nov 03 21:07:52.486761 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:52.486725    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-g44hx_a16a0bd3-5dd9-496b-9708-ee0c4091090d/istio-proxy/0.log"
Nov 03 21:07:52.903524 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:52.903390    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-tnfsk_9fa9e0c2-1bfa-4b41-b033-8d286b4c63be/istio-proxy/1.log"
Nov 03 21:07:54.426518 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:54.426454    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f7b87d695-dwgqt_d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89/router/0.log"
Nov 03 21:07:55.042703 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:07:55.042669    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-tnfsk_9fa9e0c2-1bfa-4b41-b033-8d286b4c63be/istio-proxy/0.log"
Nov 03 21:08:08.109672 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:08.109631    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:09.421229 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:09.421190    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:13.023373 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:13.023333    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"]
Nov 03 21:08:13.025967 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:13.025942    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a412c5-4kf6m"]
Nov 03 21:08:13.968284 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:13.968248    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72553983-bbaf-4120-894a-9187ef86eed8" path="/var/lib/kubelet/pods/72553983-bbaf-4120-894a-9187ef86eed8/volumes"
Nov 03 21:08:15.217385 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:08:15.217349    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:08:17.021995 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:17.021960    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:17.176195 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:17.176163    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:20.730026 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:20.729985    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hc967_ae317c95-3690-4603-a062-652384d87f55/global-pull-secret-syncer/0.log"
Nov 03 21:08:20.963259 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:20.963211    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log"
Nov 03 21:08:21.188521 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:21.188415    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-249.us-east-2.compute.internal_465e529db144367f617f146802de7125/haproxy/0.log"
Nov 03 21:08:23.022581 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:23.022539    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"]
Nov 03 21:08:23.025242 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:23.025217    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d9a142-5z7td"]
Nov 03 21:08:23.967585 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:23.967551    5659 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf494daf-114e-404a-8331-5820955a0c3f" path="/var/lib/kubelet/pods/bf494daf-114e-404a-8331-5820955a0c3f/volumes"
Nov 03 21:08:27.590888 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:27.590849    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66b59d468d-j8ndl_19730c58-ac26-42dc-bd3a-8ec0f6076264/authorino/0.log"
Nov 03 21:08:27.665177 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:27.665145    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pmwn5_a8606f49-1dc0-45c2-9720-be9731a089c4/manager/0.log"
Nov 03 21:08:27.703868 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:27.703837    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-t4sb5_856603f3-24e3-4d9e-b6dc-68dc3f57d18a/kuadrant-console-plugin/0.log"
Nov 03 21:08:29.557976 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:29.557941    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-6tkgl_0a72b034-13ca-4e05-babf-97728947b2ac/cluster-monitoring-operator/0.log"
Nov 03 21:08:29.677096 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:29.677060    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5fc5d9f9bb-s5bxr_a0edbce3-cc93-410d-b82e-27e13cb6412c/metrics-server/0.log"
Nov 03 21:08:29.852991 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:29.852917    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wvlh_146c892e-d6ef-4062-94c4-6dae80875952/node-exporter/0.log"
Nov 03 21:08:29.883312 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:29.883288    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wvlh_146c892e-d6ef-4062-94c4-6dae80875952/kube-rbac-proxy/0.log"
Nov 03 21:08:29.910296 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:29.910270    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wvlh_146c892e-d6ef-4062-94c4-6dae80875952/init-textfile/0.log"
Nov 03 21:08:30.022478 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.022447    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-v6ptx_4d03e8db-60e2-4844-94c3-78f02bc19100/kube-rbac-proxy-main/0.log"
Nov 03 21:08:30.047606 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.047575    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-v6ptx_4d03e8db-60e2-4844-94c3-78f02bc19100/kube-rbac-proxy-self/0.log"
Nov 03 21:08:30.074017 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.073988    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-v6ptx_4d03e8db-60e2-4844-94c3-78f02bc19100/openshift-state-metrics/0.log"
Nov 03 21:08:30.118528 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.118462    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da5c9072-3ba7-49da-8c2f-f81374cd3b81/prometheus/0.log"
Nov 03 21:08:30.143291 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.143267    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da5c9072-3ba7-49da-8c2f-f81374cd3b81/config-reloader/0.log"
Nov 03 21:08:30.168445 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.168421    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da5c9072-3ba7-49da-8c2f-f81374cd3b81/thanos-sidecar/0.log"
Nov 03 21:08:30.196074 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.196055    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da5c9072-3ba7-49da-8c2f-f81374cd3b81/kube-rbac-proxy-web/0.log"
Nov 03 21:08:30.221366 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.221346    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da5c9072-3ba7-49da-8c2f-f81374cd3b81/kube-rbac-proxy/0.log"
Nov 03 21:08:30.246709 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.246688    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da5c9072-3ba7-49da-8c2f-f81374cd3b81/kube-rbac-proxy-thanos/0.log"
Nov 03 21:08:30.273313 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.273291    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_da5c9072-3ba7-49da-8c2f-f81374cd3b81/init-config-reloader/0.log"
Nov 03 21:08:30.405116 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.405035    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7755d4f754-nrkkn_93b47942-ccbf-43cd-8722-c21038f906f3/telemeter-client/0.log"
Nov 03 21:08:30.431295 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.431267    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7755d4f754-nrkkn_93b47942-ccbf-43cd-8722-c21038f906f3/reload/0.log"
Nov 03 21:08:30.457627 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.457600    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7755d4f754-nrkkn_93b47942-ccbf-43cd-8722-c21038f906f3/kube-rbac-proxy/0.log"
Nov 03 21:08:30.543275 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.543239    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:30.629290 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.629257    5659 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"]
Nov 03 21:08:30.629778 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.629762    5659 cpu_manager.go:414] "containerMap: removing container" podUID="da3cc375-dadb-4c72-b71a-4ec05fec5a8d" containerName="revoke-keys"
Nov 03 21:08:30.629825 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.629780    5659 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3cc375-dadb-4c72-b71a-4ec05fec5a8d" containerName="revoke-keys"
Nov 03 21:08:30.629875 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.629844    5659 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3cc375-dadb-4c72-b71a-4ec05fec5a8d" containerName="revoke-keys"
Nov 03 21:08:30.633453 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.633429    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.640378 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.640343    5659 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c46p7\"/\"default-dockercfg-dfcj5\""
Nov 03 21:08:30.640563 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.640381    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c46p7\"/\"kube-root-ca.crt\""
Nov 03 21:08:30.640563 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.640526    5659 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c46p7\"/\"openshift-service-ca.crt\""
Nov 03 21:08:30.643832 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.643807    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"]
Nov 03 21:08:30.765976 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.765938    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82faa05e-9353-47b1-a398-fe7cb4d8876f-podres\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.766299 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.766278    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bclsr\" (UniqueName: \"kubernetes.io/projected/82faa05e-9353-47b1-a398-fe7cb4d8876f-kube-api-access-bclsr\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.766462 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.766446    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82faa05e-9353-47b1-a398-fe7cb4d8876f-proc\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.766605 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.766589    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82faa05e-9353-47b1-a398-fe7cb4d8876f-lib-modules\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.766725 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.766709    5659 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82faa05e-9353-47b1-a398-fe7cb4d8876f-sys\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.868347 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.868304    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82faa05e-9353-47b1-a398-fe7cb4d8876f-podres\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.868593 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.868369    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bclsr\" (UniqueName: \"kubernetes.io/projected/82faa05e-9353-47b1-a398-fe7cb4d8876f-kube-api-access-bclsr\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.868593 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.868401    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82faa05e-9353-47b1-a398-fe7cb4d8876f-proc\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.868593 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.868415    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82faa05e-9353-47b1-a398-fe7cb4d8876f-lib-modules\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.868593 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.868431    5659 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82faa05e-9353-47b1-a398-fe7cb4d8876f-sys\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.868593 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.868473    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82faa05e-9353-47b1-a398-fe7cb4d8876f-podres\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.868593 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.868528    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82faa05e-9353-47b1-a398-fe7cb4d8876f-lib-modules\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.868593 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.868529    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82faa05e-9353-47b1-a398-fe7cb4d8876f-proc\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.868593 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.868564    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82faa05e-9353-47b1-a398-fe7cb4d8876f-sys\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.881794 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.881768    5659 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bclsr\" (UniqueName: \"kubernetes.io/projected/82faa05e-9353-47b1-a398-fe7cb4d8876f-kube-api-access-bclsr\") pod \"perf-node-gather-daemonset-9mtnw\" (UID: \"82faa05e-9353-47b1-a398-fe7cb4d8876f\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:30.947701 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:30.947657    5659 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:31.230210 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:31.230178    5659 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"]
Nov 03 21:08:31.231740 ip-10-0-133-249 kubenswrapper[5659]: W1103 21:08:31.231715    5659 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod82faa05e_9353_47b1_a398_fe7cb4d8876f.slice/crio-bf49be23d85bcbd44f7ecd4fe500a834dccf8db34b8744ee0e9d087b9ccabc10 WatchSource:0}: Error finding container bf49be23d85bcbd44f7ecd4fe500a834dccf8db34b8744ee0e9d087b9ccabc10: Status 404 returned error can't find the container with id bf49be23d85bcbd44f7ecd4fe500a834dccf8db34b8744ee0e9d087b9ccabc10
Nov 03 21:08:31.575925 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:31.575834    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw" event={"ID":"82faa05e-9353-47b1-a398-fe7cb4d8876f","Type":"ContainerStarted","Data":"eb9e83526302ad21cac0e60d6456c0a9736bfbe32a895e314ae39c11221cc043"}
Nov 03 21:08:31.575925 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:31.575888    5659 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw" event={"ID":"82faa05e-9353-47b1-a398-fe7cb4d8876f","Type":"ContainerStarted","Data":"bf49be23d85bcbd44f7ecd4fe500a834dccf8db34b8744ee0e9d087b9ccabc10"}
Nov 03 21:08:31.575925 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:31.575919    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:31.594913 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:31.594859    5659 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw" podStartSLOduration=1.5948457390000002 podStartE2EDuration="1.594845739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:31.594077069 +0000 UTC m=+5463.748979501" watchObservedRunningTime="2026-11-03 21:08:31.594845739 +0000 UTC m=+5463.749748158"
Nov 03 21:08:32.365593 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:32.365563    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-m2pgc_114f0567-5268-473e-aec5-50869958548c/networking-console-plugin/0.log"
Nov 03 21:08:32.741520 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:32.741465    5659 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6q6gg_16b10ae2-69df-49f1-b0b3-77c98ee4cc3d/konnectivity-agent/0.log" limit=1024
Nov 03 21:08:33.271673 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:33.271634    5659 scope.go:122] "RemoveContainer" containerID="aba3a70937ea2ea597ea0db77c1bfb1decf2a2ab64134f58e7373eee994af042"
Nov 03 21:08:33.281983 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:33.281954    5659 scope.go:122] "RemoveContainer" containerID="4b53e2957208cff87e0f65a2c07f9922eac3a12b56c53ba677d5638b3c74f834"
Nov 03 21:08:33.386965 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:33.386935    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6k7gv_764bdfb5-060a-4657-b9e0-254f1795eea8/console-operator/0.log"
Nov 03 21:08:34.153364 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:34.153335    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7b6799d47f-t448x_da47f42e-3aa5-4f3c-9b56-549760b8bab3/console/0.log"
Nov 03 21:08:34.190579 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:34.190552    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-8dg79_13c92206-9485-47b9-a056-dcb8ec3588c4/download-server/0.log"
Nov 03 21:08:34.942907 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:34.942875    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-jhb7v_ece7bfca-bf0e-4422-ad06-1df2ca9a08bf/volume-data-source-validator/0.log"
Nov 03 21:08:36.363918 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:36.363886    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5xsml_674ad509-03fd-4f35-bc70-3a3602457de9/dns/0.log"
Nov 03 21:08:36.388562 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:36.388537    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5xsml_674ad509-03fd-4f35-bc70-3a3602457de9/kube-rbac-proxy/0.log"
Nov 03 21:08:36.554458 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:36.554430    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5ktfb_e79d1e48-bb87-4a1b-9b88-95a1551e959c/dns-node-resolver/0.log"
Nov 03 21:08:37.379125 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:37.379093    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mmxqm_ad627830-62cd-4f5b-bb63-76b5944b91ff/node-ca/0.log"
Nov 03 21:08:37.592000 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:37.591970    5659 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-9mtnw"
Nov 03 21:08:39.762318 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:39.762286    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fxxs9_aa079182-95a7-4454-be04-45f06a41088f/manager/0.log"
Nov 03 21:08:40.533242 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:40.533202    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8qktw_fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a/discovery/0.log"
Nov 03 21:08:40.733161 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:40.733127    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qkjdd_950e6c44-5e74-4c6c-9c38-0712a81af85a/manager/0.log"
Nov 03 21:08:40.912074 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:40.911994    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-76gw5_0f42d32f-c8f1-48b4-807a-76636151cb72/server/0.log"
Nov 03 21:08:40.958125 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:40.958095    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79ddfb4884-vtwjp_727ca680-846f-4226-b4ee-1dad5f785224/kube-auth-proxy/0.log"
Nov 03 21:08:41.141312 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:41.141276    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-g44hx_a16a0bd3-5dd9-496b-9708-ee0c4091090d/istio-proxy/0.log"
Nov 03 21:08:41.217067 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:41.217004    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-tnfsk_9fa9e0c2-1bfa-4b41-b033-8d286b4c63be/istio-proxy/0.log"
Nov 03 21:08:41.222876 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:41.222851    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-tnfsk_9fa9e0c2-1bfa-4b41-b033-8d286b4c63be/istio-proxy/1.log"
Nov 03 21:08:41.223724 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:41.223690    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-564sz_67be1eda-e12d-4a81-9785-5c5c8c3989d2/proxy/0.log"
Nov 03 21:08:41.548042 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:41.547969    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f7b87d695-dwgqt_d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89/router/0.log"
Nov 03 21:08:43.033786 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:43.033746    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fxxs9_aa079182-95a7-4454-be04-45f06a41088f/manager/0.log"
Nov 03 21:08:44.041850 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:44.041818    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qkjdd_950e6c44-5e74-4c6c-9c38-0712a81af85a/manager/0.log"
Nov 03 21:08:44.171396 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:44.171363    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-76gw5_0f42d32f-c8f1-48b4-807a-76636151cb72/server/0.log"
Nov 03 21:08:44.509209 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:44.509174    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-564sz_67be1eda-e12d-4a81-9785-5c5c8c3989d2/proxy/0.log"
Nov 03 21:08:45.156734 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:45.156700    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8qktw_fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a/discovery/0.log"
Nov 03 21:08:45.169430 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:45.169408    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pqv25_74569270-9489-4c9f-b51e-d819081baa5d/serve-healthcheck-canary/0.log"
Nov 03 21:08:45.188017 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:45.187997    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79ddfb4884-vtwjp_727ca680-846f-4226-b4ee-1dad5f785224/kube-auth-proxy/0.log"
Nov 03 21:08:45.216422 ip-10-0-133-249 kubenswrapper[5659]: E1103 21:08:45.216391    5659 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kgm4l" containerName="extractor"
Nov 03 21:08:45.285408 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:45.285383    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-g44hx_a16a0bd3-5dd9-496b-9708-ee0c4091090d/istio-proxy/0.log"
Nov 03 21:08:45.334666 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:45.334637    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-tnfsk_9fa9e0c2-1bfa-4b41-b033-8d286b4c63be/istio-proxy/0.log"
Nov 03 21:08:45.338814 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:45.338787    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-tnfsk_9fa9e0c2-1bfa-4b41-b033-8d286b4c63be/istio-proxy/1.log"
Nov 03 21:08:45.490862 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:45.490832    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f7b87d695-dwgqt_d4a9f3ab-0fb9-4bec-9bd8-e68dfbf08a89/router/0.log"
Nov 03 21:08:45.743984 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:45.743888    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fxxs9_aa079182-95a7-4454-be04-45f06a41088f/manager/0.log"
Nov 03 21:08:45.980640 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:45.980590    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kgm4l_e50201a6-9ca6-4ff0-9e55-cc324745331e/kube-rbac-proxy/0.log"
Nov 03 21:08:46.009437 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:46.009365    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kgm4l_e50201a6-9ca6-4ff0-9e55-cc324745331e/exporter/0.log"
Nov 03 21:08:46.044291 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:46.044263    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kgm4l_e50201a6-9ca6-4ff0-9e55-cc324745331e/extractor/0.log"
Nov 03 21:08:46.510393 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:46.510358    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qkjdd_950e6c44-5e74-4c6c-9c38-0712a81af85a/manager/0.log"
Nov 03 21:08:46.687841 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:46.687808    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-76gw5_0f42d32f-c8f1-48b4-807a-76636151cb72/server/0.log"
Nov 03 21:08:47.010510 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:47.010443    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-564sz_67be1eda-e12d-4a81-9785-5c5c8c3989d2/proxy/0.log"
Nov 03 21:08:48.918376 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:48.918342    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8464c976fd-cpxbn_4678c42e-c435-4d42-b40e-f75257db51aa/manager/0.log"
Nov 03 21:08:48.947019 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:48.946987    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8464c976fd-cpxbn_4678c42e-c435-4d42-b40e-f75257db51aa/copy-manifests/0.log"
Nov 03 21:08:49.973010 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:49.972980    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8qktw_fc6d8d2d-637f-4f1a-9834-dffb9dbc2c6a/discovery/0.log"
Nov 03 21:08:50.005493 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:50.005451    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79ddfb4884-vtwjp_727ca680-846f-4226-b4ee-1dad5f785224/kube-auth-proxy/0.log"
Nov 03 21:08:50.023552 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:50.023520    5659 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"]
Nov 03 21:08:50.026866 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:50.026836    5659 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bce983d-sbgcs"]
Nov 03 21:08:50.110135 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:50.110103    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-g44hx_a16a0bd3-5dd9-496b-9708-ee0c4091090d/istio-proxy/0.log"
Nov 03 21:08:50.155270 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:50.155236    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-tnfsk_9fa9e0c2-1bfa-4b41-b033-8d286b4c63be/istio-proxy/0.log"
Nov 03 21:08:50.159305 ip-10-0-133-249 kubenswrapper[5659]: I1103 21:08:50.159284    5659 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-tnfsk_9fa9e0c2-1bfa-4b41-b033-8d286b4c63be/istio-proxy/1.log"